An intelligent lock unlocking method and system based on multi-modal biometric feature fusion

Abstract

The application discloses a kind of based on multi-modal biological feature fusion intelligent lock unlocking method and system, it is related to identity recognition technical field, the method includes: extracting user face global structure feature vector, fingerprint image sequence and capacitance response time series signal, output face recognition confidence score and fingerprint matching degree score;Extracting capacitance-time dynamic curve, frequency domain transformation is carried out, and frequency spectrum distribution characteristic vector is obtained, the deformation recovery rate of ridge line in initial stage is extracted, and mechanical creep characteristic vector is obtained;Based on pre-trained living body classifier, output living body confidence score;Based on environmental perception data, obtain comprehensive security score, if higher than preset threshold, then execute unlocking action.The technical problem that single biological feature recognition of existing intelligent lock is easily counterfeited and lacks living body detection capability, multi-modal fusion mode is fixed and cannot adapt to complex and changeable use environment, leading to security and user experience difficult to take into account is solved.

Description

Technical Field

[0001] This application relates to the field of identity recognition technology, specifically to a smart lock unlocking method and system based on multimodal biometric fusion. Background Technology

[0002] Smart locks are becoming the mainstream choice for home locks due to their ease of use and the fact that they do not require keys. To ensure unlocking security, most smart locks use biometric identification technology for identity verification, including facial recognition unlocking and fingerprint recognition unlocking.

[0003] Among them, single biometric identification schemes have obvious defects: facial recognition is easily affected by ambient light, and the recognition accuracy drops significantly in backlight and low light environments, and it is easily deceived by spoofing attacks such as photos and videos; single fingerprint recognition is easily affected by the dryness or wetness of the fingers and the wear and tear of the skin, and the success rate of recognizing blurry fingerprints is low. It is also at risk of being spoofed and cracked by fake fingerprints made of materials such as silicone and resin.

[0004] While some solutions have begun to adopt multi-feature fusion schemes combining face and fingerprint, they simply splice together two static identity features without setting up a dedicated detection step for liveness spoofing attacks, thus failing to effectively identify non-liveness impersonation attacks. Furthermore, most existing multi-feature fusion schemes employ fixed-weighted fusion strategies, which cannot adapt to the fluctuations in recognition reliability of different features under different environments, and are prone to false recognition or excessively high false rejection rates in complex environments. Summary of the Invention

[0005] This application provides a smart lock unlocking method and system based on multimodal biometric fusion, which solves the technical problems of existing smart locks where single biometric recognition is easily counterfeited and lacks liveness detection capabilities, and the fixed multimodal fusion method cannot adapt to complex and ever-changing usage environments, making it difficult to balance security and user experience.

[0006] The technical solution to the above-mentioned technical problems in this application is as follows: In a first aspect, this application provides a smart lock unlocking method based on multimodal biometric fusion, the method comprising: Collect user facial images, extract global facial structure feature vectors, match them with a local face feature database, and output a face recognition confidence score. If the face recognition confidence score is higher than the face confidence threshold, the fingerprint image sequence and capacitive response timing signal during the user's finger pressing process are collected simultaneously, the fingerprint image sequence is matched with ridge features, and the fingerprint matching score is output. In response to the fingerprint matching score exceeding the fingerprint confidence threshold, the capacitance-time dynamic curve of the entire finger pressing process is extracted from the capacitance response time sequence signal, the capacitance-time dynamic curve is transformed in the frequency domain to obtain the spectral distribution feature vector, and the ridge deformation recovery rate at the initial stage of finger pressing is extracted from the fingerprint image sequence to obtain the mechanical creep feature vector. The spectral distribution feature vector and the mechanical creep feature vector are input into a pre-trained liveness classifier, which outputs a liveness confidence score. Based on environmental perception data, the face recognition confidence score, the fingerprint matching score, and the liveness confidence score are adaptively weighted and fused to generate a comprehensive security score. If the comprehensive security score is higher than a preset threshold, the unlocking action is performed; otherwise, the unlocking is refused and an alarm is triggered.

[0007] Secondly, this application provides a smart lock unlocking system based on multimodal biometric fusion, comprising: The face feature matching module is used to collect user facial images, extract global facial structure feature vectors, match them with the local face feature database, and output a face recognition confidence score. The fingerprint feature matching module is used to simultaneously collect the fingerprint image sequence and capacitive response timing signal during the user's finger pressing process if the face recognition confidence score is higher than the face confidence threshold, perform ridge feature matching on the fingerprint image sequence, and output the fingerprint matching score. The capacitance-mechanical data acquisition module is used to extract the capacitance-time dynamic curve of the entire finger pressing process from the capacitance response time sequence signal in response to the fingerprint matching score exceeding the fingerprint confidence threshold, perform frequency domain transformation on the capacitance-time dynamic curve to obtain the spectral distribution feature vector, and extract the ridge deformation recovery rate at the initial stage of finger pressing from the fingerprint image sequence to obtain the mechanical creep feature vector. The liveness detection module is used to input the spectral distribution feature vector and the mechanical creep feature vector into a pre-trained liveness classifier and output a liveness confidence score. The unlocking execution module is used to adaptively weight and fuse the face recognition confidence score, the fingerprint matching score, and the liveness confidence score based on environmental perception data to generate a comprehensive security score. If the comprehensive security score is higher than a preset threshold, the unlocking action is executed; otherwise, the unlocking is refused and an alarm is triggered.

[0008] This application provides one or more technical solutions, which have at least the following technical effects or advantages: This application provides a smart lock unlocking method and system based on multimodal biometric fusion. First, building upon static identity feature matching using both facial and fingerprint data, a dynamic liveness detection mechanism based on capacitive response timing signals is introduced. By extracting the spectral distribution feature vector and mechanical creep feature vector during finger pressing, a genuine live finger can be effectively distinguished from a fake one. Simultaneously, by combining the facial micro-vibration periodic signal with the time phase difference at the start of finger pressing, a multimodal physiological synchronicity constraint is constructed, enhancing the defense against complex deception attacks such as simulated fakes. Furthermore, the fusion weights of each biometric score are dynamically adjusted based on ambient light intensity and finger skin humidity, achieving adaptive optimization of the recognition strategy in complex environments. This ensures high security while reducing the false rejection rate, balancing user experience and unlocking reliability.

[0009] Through the above technical solution, this application solves the problems of existing smart locks, such as the susceptibility of single biometric recognition to counterfeiting and the lack of liveness detection capabilities, and the inability of fixed multimodal fusion methods to adapt to complex and changing usage environments. It can effectively resist various common deception attacks such as photos, videos, and fake fingerprints. Even in abnormal lighting environments such as low light and backlight, as well as poor fingerprint collection conditions such as excessively dry or wet fingers, it can still maintain a stable recognition accuracy rate. This not only improves the overall security of smart lock unlocking, but also avoids the problem of excessively high false rejection rate in complex environments, balancing security and ease of use, and is suitable for the promotion and application of smart locks in both home and commercial settings. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating a smart lock unlocking method based on multimodal biometric fusion provided in an embodiment of this application; Figure 2 This is a schematic diagram of the process of outputting a liveness confidence score in a smart lock unlocking method based on multimodal biometric fusion provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a smart lock unlocking system based on multimodal biometric fusion provided in an embodiment of this application.

[0012] The components represented by each number in the attached diagram are explained below: The system includes a face feature matching module 11, a fingerprint feature matching module 12, a capacitance-mechanical data acquisition module 13, a liveness detection module 14, and an unlocking execution module 15. Detailed Implementation

[0013] This application provides a smart lock unlocking method and system based on multimodal biometric fusion, which addresses the technical problems of existing smart locks where single biometric recognition is easily counterfeited and lacks liveness detection capabilities, and the fixed multimodal fusion method cannot adapt to complex and changing usage environments, making it difficult to balance security and user experience.

[0014] The smart lock structure provided by this invention includes a face acquisition unit, a fingerprint acquisition unit, a signal processing unit, a liveness classifier, and a lock control execution unit. The face acquisition unit and the fingerprint acquisition unit are connected to the signal processing unit via MIPI and SPI interfaces, respectively. The signal processing unit has a built-in or external storage module for storing a local face feature library, a local fingerprint feature library, and pre-trained liveness classifier model parameters.

[0015] In a preferred embodiment, a wide-angle camera is embedded in the smart lock panel. This camera has video intercom and image acquisition functions, eliminating the need for additional dedicated face acquisition hardware. The fingerprint acquisition unit uses a commercially available capacitive fingerprint sensor chip. While outputting the fingerprint image, this chip can obtain the capacitance response value of each pixel by reading its internal register. After the smart lock is started, the signal processing unit sequentially activates the face acquisition unit and the fingerprint acquisition unit according to a preset workflow, and synchronously records the capacitance response timing signal during the fingerprint acquisition process.

[0016] Example 1, as Figure 1 As shown in the figure, this application provides a smart lock unlocking method based on multimodal biometric fusion, including: S10: Collect user facial images, extract global facial structure feature vectors, match them with the local face feature database, and output face recognition confidence score; In this embodiment, a wide-angle camera is used to automatically trigger image acquisition when the user approaches the smart lock. Infrared supplementary lighting is used to achieve clear imaging in low-light environments, extract facial key points and global structural features, generate a facial feature vector, and perform similarity matching with the locally stored registered user feature library to output a facial recognition confidence score.

[0017] Specifically, step S10 in the method includes: Perform face detection and alignment on the facial image, and crop out the face region; The global structural feature vector of the face region is extracted using a pre-trained lightweight convolutional neural network; Calculate the cosine similarity between the global structural feature vector and each template feature vector in the local face feature library, and output the maximum similarity value as the face recognition confidence score; If the facial recognition confidence score is lower than the facial confidence threshold, the smart lock's fill light will be turned on and the facial image will be re-acquired. After a preset number of consecutive failures, the lock will enter a locked state.

[0018] In this embodiment, firstly, when the user approaches the smart lock, the infrared human body sensor is triggered to wake up the smart lock. The signal processing unit controls the camera to capture the user's facial image. Since the smart lock is usually installed 0.3 to 0.8 meters away from the user's face, the camera's focal length and field of view should be adapted to this distance range to ensure that the face occupies no less than 30% of the image.

[0019] Since the original facial images may have problems such as uneven lighting and noise interference, the images are first preprocessed. Specifically, median filtering is used to remove salt-and-pepper noise, histogram equalization is used to improve image contrast, and edge-preserving filtering is used to smooth the image while preserving key facial texture details.

[0020] Secondly, the preprocessed image is input into the face detection module, and the existing multi-task cascaded convolutional neural network is used for face detection and key point localization. The multi-task cascaded convolutional neural network achieves coarse-to-fine face detection through three cascaded networks, namely P-Net, R-Net, and O-Net, and outputs the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth, and the right corner of the mouth.

[0021] Based on the detected keypoint coordinates, an affine transformation is performed on the original facial image to align the face to a standard pose. Specifically, the five detected keypoints are mapped to preset standard template keypoints, the affine transformation matrix is ​​calculated, and then the facial region in the image is uniformly adjusted to a fixed size, such as 112 pixels × 112 pixels, through the affine transformation to eliminate the influence of pose changes on subsequent feature extraction. Through the alignment operation, the facial images of different users are in a consistent geometric coordinate system before entering the feature extraction network, which helps to improve the comparability of features.

[0022] Next, the aligned face image is input into a pre-trained lightweight convolutional neural network. In a preferred embodiment, the MobileFaceNet network structure is used. This network compresses the number of parameters to about 1.2 megabytes through depthwise separable convolutions, making it suitable for real-time operation on the embedded processor of a smart lock. The network is pre-trained on a dataset so that it can map the input face image into a fixed-length feature vector. The last layer of the network outputs a 128-dimensional or 256-dimensional global structural feature vector, which encodes the global geometric structure and texture distribution information of the face, such as facial morphological features like eye spacing, nose bridge height, and cheekbone contour.

[0023] Furthermore, the extracted global structural feature vector is compared with each template feature vector pre-stored in the local face feature library to calculate the cosine similarity. The cosine similarity ranges from -1 to 1. The closer the value is to 1, the more similar the two feature vectors are. The maximum similarity value is mapped to the interval [0,1] as the face recognition confidence score.

[0024] For example, in a preferred embodiment, the ReLU truncation method is used, that is, the final face recognition confidence score is max(0, the maximum cosine similarity value between the current user and all templates in the local face feature library). If the maximum value is negative, it usually means that the user has never registered and the facial features are negatively correlated with all templates, then the confidence score is 0, that is, no trust at all.

[0025] The local facial feature database is pre-established during the user registration phase. During registration, users follow prompts to capture multiple frames of facial images with different poses and expressions in front of the camera. The global structural feature vectors of each frame are extracted, and after averaging or clustering, a standard facial template for the user is generated and stored locally in association with the user's identity. The advantage of local storage is that it does not rely on the network, avoiding the privacy leakage risks associated with cloud transmission.

[0026] For example, the formula for calculating cosine similarity is: sim(F1,F2)= , where F1 is the currently extracted global structural feature vector, F2 is the template feature vector, · represents the vector dot product, and |F1| and |F2| represent the L2 norms of the two vectors respectively.

[0027] If the facial recognition confidence score is lower than the preset facial confidence threshold, the initial facial identity screening is deemed unsuccessful. The facial confidence threshold can be set to 0.60. In this case, the fill light on the smart lock panel is turned on to increase the ambient light intensity, and the facial image is re-acquired and matched again. After a preset number of consecutive failures, such as 3 or 5 times, the smart lock enters a locked state and will not respond to unlocking requests for a preset time, such as 15 seconds, to prevent brute-force attacks.

[0028] S20: If the face recognition confidence score is higher than the face confidence threshold, the fingerprint image sequence and capacitive response timing signal during the user's finger pressing process are collected simultaneously, the fingerprint image sequence is matched with ridge features, and the fingerprint matching score is output. In this embodiment of the application, if the face recognition confidence score is higher than the face confidence threshold, it is determined that the user's initial identity verification is successful, and then the fingerprint collection unit is activated to enter the second biometric verification stage.

[0029] At this point, the user needs to press their finger on the fingerprint sensor area, and simultaneously perform the acquisition of multiple frames of fingerprint images and read the capacitance values ​​of each pixel output by the internal register of the fingerprint sensor to form a fingerprint image sequence. A capacitance response timing signal synchronized with time is also constructed. Then, the fingerprint image sequence is matched with ridge features to output a fingerprint matching score.

[0030] This process involves simultaneously acquiring fingerprint image sequences and capacitive response timing signals during the user's finger pressing process, performing ridge feature matching on the fingerprint image sequences, and outputting a fingerprint matching score, including: Based on the capacitive fingerprint sensor, multiple frames of fingerprint images are continuously acquired from the process of finger contact to full pressing at a preset sampling frame rate to form the fingerprint image sequence, and the capacitance response value corresponding to each frame of fingerprint image is recorded synchronously to form the capacitance response timing signal. Select a frame of fingerprint image from the fingerprint image sequence at the moment of stable pressure, extract the minutiae features, and perform minutiae matching with the local fingerprint feature library. Output the matching score as the fingerprint matching degree score.

[0031] In this embodiment of the application, when the face recognition confidence score is higher than the face confidence threshold, the fingerprint identity verification stage is entered, and the user presses their finger on the capacitive fingerprint sensor of the smart lock.

[0032] Specifically, the working principle of a capacitive fingerprint sensor is as follows: The sensor surface is integrated with a tiny array of capacitor plates. When a finger touches the sensor surface, the distance between the ridge and valley of the finger, that is, the raised part and the recessed part, is different from the sensor surface, resulting in differences in the capacitance value of each pixel, thus forming a fingerprint grayscale image. At the same time, when the sensor outputs each frame of fingerprint image, it can simultaneously output the average capacitance response value of each pixel corresponding to that frame of image.

[0033] Furthermore, the entire process of a finger from contacting the sensor surface to complete and stable pressing is continuously captured at a preset sampling frame rate. For example, the preset sampling frame rate can be set to 60 frames per second or higher to ensure that the rapid deformation process at the beginning of pressing can be captured, and fingerprint image frames arranged in chronological order are obtained to form a fingerprint image sequence. Each fingerprint image carries a timestamp tag, which is convenient for subsequent alignment with the capacitive response timing signal.

[0034] While acquiring fingerprint image sequences, the overall capacitance response value corresponding to each frame of fingerprint image is synchronously recorded through the register read interface of the fingerprint sensor chip. For example, the sum or average value of the capacitance values ​​of each pixel is recorded and arranged in the order of acquisition time to form a capacitance response timing signal. This signal is the data basis for capacitance frequency domain analysis in subsequent steps.

[0035] Furthermore, a frame of fingerprint image at the moment when the finger is pressed stably is selected from the above fingerprint image sequence for minutiae matching. The criterion for judging the moment of pressing stably can be set as follows: when the average ridge width and the rate of change of image contrast of multiple consecutive fingerprint images are both lower than a preset rate of change threshold, the finger is considered to have reached a stable state.

[0036] The selected stable frame fingerprint images are subjected to image enhancement processing, including orientation field estimation, Gabor filtering enhancement, binarization and thinning, to extract the minutiae features of the fingerprint. Minutiae include ridge endpoints and ridge bifurcation points, i.e., the termination position of the ridge and the position where the ridge splits into two. By calculating the position coordinates, orientation angle and type information of the minutiae, the fingerprint minutiae feature vector is constructed.

[0037] The extracted minutiae feature vector is matched with the fingerprint template of the user in the local fingerprint feature library to calculate the matching score. During the matching process, the minutiae to be identified are first aligned with the minutiae of the template, that is, by rotating and translating to make as many minutiae as possible overlap. Then, the ratio of the number of matched minutiae pairs to the total number of minutiae is calculated. Combined with the local orientation consistency of each minutiae, the fingerprint matching score is obtained. The score range is normalized to between 0 and 1.

[0038] If the fingerprint matching score is lower than the preset fingerprint confidence threshold, the identity verification is deemed unsuccessful, the unlock is refused, and an alarm is triggered as appropriate. The alarm may be a short beep from a buzzer or an abnormal unlock notification sent to the bound mobile terminal. The preset fingerprint confidence threshold can be set to 0.55.

[0039] S30: In response to the fingerprint matching score exceeding the fingerprint confidence threshold, extract the capacitance-time dynamic curve of the entire finger pressing process from the capacitance response time sequence signal, perform frequency domain transformation on the capacitance-time dynamic curve to obtain the spectral distribution feature vector, and extract the ridge deformation recovery rate at the initial stage of finger pressing from the fingerprint image sequence to obtain the mechanical creep feature vector. In this embodiment, when the fingerprint matching degree exceeds a preset threshold, two types of features required for liveness detection can be extracted based on the acquired capacitance response time-series signal and fingerprint image sequence: the capacitance-time dynamic curve and the ridge deformation recovery rate at the initial stage of finger pressing. The obtained capacitance-time dynamic curve is then subjected to a fast Fourier transform to convert the capacitance change signal in the time domain into a spectral distribution in the frequency domain. The amplitude components of different frequency bands are extracted to construct a spectral distribution feature vector. This vector can reflect the frequency composition differences in the capacitance change process and distinguish the dynamic change patterns of real live objects and spurious objects.

[0040] Among them, the capacitance-time dynamic curve records the dynamic law of the overall capacitance value changing with finger pressure throughout the entire process from contact, pressing to stabilization. The skin of a real living finger is elastic, and the capacitance value will show a gradual change from the initial contact to a rapid increase, and then tend to stabilize after the skin is squeezed. In contrast, the prosthetic fingerprint is usually made of rigid material, and the capacitance value changes abruptly without a smooth gradual process.

[0041] The initial ridge deformation recovery rate under finger pressure is determined by the mechanical creep characteristics of living skin. The skin elasticity varies among individuals, but all conform to the creep law of soft tissue. However, prosthetic materials do not have this biological characteristic. Therefore, mechanical creep feature vectors are extracted to complete the construction of two types of living features.

[0042] Specifically, in response to the fingerprint matching score exceeding the fingerprint confidence threshold, the capacitance-time dynamic curve of the entire finger pressing process is extracted from the capacitance response time-series signal. The capacitance-time dynamic curve is then subjected to frequency domain transformation to obtain a spectral distribution feature vector, including: The capacitor response timing signal is preprocessed by denoising and normalization to obtain the capacitor-time dynamic curve; A fast Fourier transform is performed on the capacitance-time dynamic curve to extract the spectral amplitude distribution in the frequency band from 0.5 Hz to 15 Hz. The spectral amplitude distribution is integrated according to a preset frequency band interval, and the integrated energy value of each frequency band interval is used to form the spectral distribution feature vector; The preset frequency band range includes at least a low-frequency band that reflects involuntary finger muscle tremors and a mid-frequency band that reflects tissue elastic resonance.

[0043] In this embodiment, the capacitance response timing signal is preprocessed by denoising and normalization. Denoising is performed by sliding window mean filtering or median filtering, with a window size of, for example, 5 sampling points, to eliminate high-frequency random noise. Normalization maps the capacitance response value at each time point to the range of 0 to 1 to eliminate the influence of differences in the absolute value of capacitance amplitude caused by different user finger sizes and different pressure levels on subsequent analysis.

[0044] After preprocessing, a capacitance-time dynamic curve is constructed with time as the horizontal axis and capacitance response value as the vertical axis. This curve reflects the change of capacitance response of finger tissue under mechanical load over time from the moment the finger touches the sensor until the pressure stabilizes.

[0045] Secondly, a Fast Fourier Transform (FFT) is performed on the capacitance-time dynamic curve to transform it from the time domain to the frequency domain, and the spectral amplitude distribution in the 0.5 Hz to 15 Hz frequency band is extracted. The selection of this frequency band is based on the following criteria: The 0.5 Hz to 4 Hz frequency band mainly reflects the periodic changes in blood flow pulses related to heart rate, as well as low-frequency fluctuations caused by autonomic nervous system regulation; the 4 Hz to 15 Hz frequency band mainly reflects the physiological unconscious shaking of finger muscles during the pressing process, i.e., physiological tremor, as well as the elastic resonance response of finger tissues under external force.

[0046] The extracted spectral amplitude distribution is integrated according to a preset frequency band interval. In a preferred embodiment, the preset frequency band interval includes at least the following two intervals: Low frequency band: 0.5 Hz to 4 Hz, mainly reflecting blood flow-related periodic physiological signals; Mid-frequency band: 8 Hz to 13 Hz, mainly reflecting the frequency distribution of involuntary muscle twitching.

[0047] The spectral distribution feature vector is constructed using the integrated energy values ​​across each frequency band. The blood flow pulse signal of a living finger maintains a high level of integrated energy in the low-frequency band, while imitation materials without living fingers lack this blood flow signal, resulting in integrated energy values ​​close to zero in the low-frequency band. Furthermore, the involuntary tremors of a living finger caused by dynamic muscle regulation produce a specific energy distribution pattern in the mid-frequency band, whereas imitation materials such as silicone, due to their homogeneous structure, lack this complex physiological tremor energy distribution.

[0048] Further, the ridge deformation recovery rate at the initial stage of finger pressure is extracted from the fingerprint image sequence to obtain a mechanical creep feature vector, including: Extract the first N frames of fingerprint images from the fingerprint image sequence after the finger touches the surface of the smart lock, where N is a preset positive integer; Calculate the mean ridge width and ridge direction distribution entropy frame by frame for the first N frames of fingerprint images; The rate of change of the average ridge width over time is taken as the ridge deformation recovery rate. The mechanical creep feature vector is formed by combining the ridge deformation recovery rate and the trend of the ridge direction distribution entropy.

[0049] In this embodiment, fingerprint images of the first N frames after a finger contacts the surface of the smart lock are extracted from the fingerprint image sequence, where N is a preset positive integer. In a preferred embodiment, when the sampling frame rate is 60 frames per second, N is 10 to 15 frames, corresponding to a time window of approximately 0.17 to 0.25 seconds after the finger contacts the sensor surface. This time window covers the rapid mechanical process from the finger's contact with the sensor surface to the beginning of significant deformation and gradual recovery.

[0050] Secondly, for each captured fingerprint image frame, the mean ridge width and ridge direction distribution entropy are calculated frame by frame. Specifically, the fingerprint image is binarized to separate the ridges and valleys. The ridges are scanned along the direction perpendicular to the ridges, the width of the ridges at each sampling point is measured, and the average of the ridge width values ​​at all sampling points is calculated to obtain the mean ridge width of the image frame.

[0051] For example, to calculate the average ridge width, a local window of 15 pixels in length can be selected segment by segment along the ridge direction. A straight line perpendicular to the ridge direction can be drawn in each window, and the number of pixels belonging to the ridge region on the straight line can be counted. This number represents the ridge width at that position. The average of the widths at all positions is the average ridge width of the entire frame image.

[0052] The fingerprint image is divided into several local blocks, such as 16 pixels × 16 pixels. Within each block, the main direction of the ridges is calculated through gradient analysis. The direction distribution histogram of all local blocks is statistically analyzed. The 0 to 180 degrees is discretized into 8 or 16 direction intervals. Then, the information entropy value of the histogram is calculated. The direction distribution entropy reflects the degree of order of the fingerprint ridge direction. That is, when the finger is deformed by pressure, the ridge direction distribution tends to be dispersed and the entropy value increases; when the deformation is restored, the ridges return to the original regular direction and the entropy value decreases.

[0053] Furthermore, the rate of change of the average ridge width over time is used as the ridge deformation recovery rate. Specifically, the average ridge width of each frame in the first N frames is linearly fitted or subjected to first-order difference operation to obtain its slope over time, which is used as the deformation recovery rate. The skin of a living finger is viscoelastic, and the ridge will deform rapidly in the initial stage of pressing, that is, the width will increase. Then it will gradually recover due to the elasticity of the skin. In contrast, non-living imitation materials are mostly plastic materials or elastomers, and their deformation-recovery curves are significantly different from those of living tissue.

[0054] The calculated ridge deformation recovery rate and the trend of ridge direction distribution entropy are combined to form a mechanical creep feature vector, which encodes the mechanical response characteristics of finger skin during contact and pressure.

[0055] S40: Input the spectral distribution feature vector and the mechanical creep feature vector into a pre-trained liveness classifier and output a liveness confidence score; In the embodiments of this application, such as Figure 2 As shown, the spectral distribution feature vector and the mechanical creep feature vector are normalized and then concatenated into a fused feature vector to eliminate the influence of the difference in the dimensions of different features on the classifier. The fused feature vector is input into a pre-trained liveness classifier, which outputs a probability value between 0 and 1 as a liveness confidence score. The closer the score is to 1, the closer the input signal is to the feature pattern of a live finger, and the closer the score is to 0, the closer it is to the feature pattern of counterfeit materials.

[0056] Specifically, step S40 in the method includes: The spectral distribution feature vector and the mechanical creep feature vector are normalized and then concatenated to form a fused feature vector; The fused feature vector is input into a pre-trained lightweight neural network classifier, and the probability value output by the liveness classifier is used as the liveness confidence score. The training samples for the pre-trained liveness classifier include capacitive response data of real live fingers and capacitive response data of various non-liveness imitation materials.

[0057] In this embodiment, firstly, z-score normalization is performed on the two types of features respectively, mapping each dimension of the features to a distribution with a mean of 0 and a variance of 1. Then, the normalized spectral distribution feature vector and the mechanical creep feature vector are concatenated along the feature dimension to obtain a complete fused feature vector. This feature integrates three types of biological information: the physiological circulatory characteristics of the finger, the muscle and nerve activity characteristics, and the skin mechanical characteristics, which can more comprehensively distinguish between real living fingerprints and fake fingerprints.

[0058] Secondly, the fused feature vectors are input into a pre-trained lightweight neural network classifier. This classifier adopts a simple fully connected network architecture with few parameters, making it suitable for deployment in low-computing-power embedded devices on the smart lock side. The probability value between 0 and 1 output by the classifier is the liveness confidence score. When training this classifier, the training samples cover the collection data of various common counterfeit materials such as silicone, rubber, and 3D printed fake fingerprints, as well as the liveness data of real users of different ages and skin types, ensuring that the classifier has sufficient generalization ability.

[0059] Furthermore, before inputting the spectral distribution feature vector and the mechanical creep feature vector into the pre-trained liveness classifier, the method further includes: Obtain the pixel optical flow change sequence of the cheek region in the facial image, and extract the facial micro-vibration periodic signal; The finger pressing start time in the capacitive response timing signal is acquired synchronously, and the time phase difference between the facial micro-vibration period signal and the finger pressing start time is calculated. The time phase difference is used as a synchronization constraint factor, and it is concatenated and fused with the spectral distribution feature vector and the mechanical creep feature vector, and then input into the pre-trained liveness classifier. If the time phase difference exceeds the preset human physiological synchronization time window, it is directly determined that there is a mismatch.

[0060] In this embodiment of the application, the liveness detection process also includes face-finger spatiotemporal linkage detection as a pre-constraint condition. This mechanism utilizes the physiological synchronicity between the face and fingers of the same living organism to further improve the reliability of liveness detection.

[0061] First, in the user's facial image sequence captured by the camera, because the skin on the cheek is thin and close to the superficial arteries of the face, the optical changes caused by the heartbeat are more obvious. Therefore, the user's cheek area is selected as the region of interest. The pixel optical flow changes are calculated for multiple consecutive frames of the region of interest, that is, the tiny displacement of the same pixel between two adjacent frames is analyzed. Because the periodic fluctuation amplitude of the facial skin caused by the heartbeat is extremely small, usually on the order of micrometers, it is manifested in the image as a weak periodic change in pixel grayscale.

[0062] Bandpass filtering is applied to the optical flow variation sequence, for example, in the frequency band of 0.5 Hz to 3.0 Hz, to filter out noise and high-frequency interference, and then a signal with periodic regularity is extracted, namely the facial micro-vibration periodic signal. The frequency of this signal is consistent with the user's heart rate, usually between 60 and 100 times per minute, with a frequency band between 1.0 Hz and 1.67 Hz.

[0063] Furthermore, the finger pressing start time in the capacitance response timing signal recorded in the aforementioned steps is obtained synchronously. This start time is defined as the moment when the capacitance response value first exceeds the preset contact threshold. The time phase difference between the facial micro-vibration period signal and the finger pressing start time is calculated.

[0064] Specifically, within a preset time window before the start of finger pressing, such as the first 3 seconds, the waveform of the facial micro-vibration cycle signal is analyzed to find the most recent peak moment of facial micro-vibration caused by heartbeat; the time interval between this peak moment and the start of finger pressing is calculated as the time phase difference.

[0065] Furthermore, when a living organism presses its fingers forcefully, the excitation of the sympathetic nervous system causes a small change in heart rate. This change is transmitted from the heart to the facial blood vessels and finger muscles with a fixed physiological delay, which is usually in the range of 200 to 600 milliseconds.

[0066] If the calculated time phase difference exceeds the preset human physiological synchronization time window, such as the confidence interval of 200 milliseconds to 600 milliseconds, it indicates that the synchronization relationship between the signals of the face and the fingers on the time axis does not conform to the physiological laws of living organisms. The system determines that the face and the fingers do not belong to the same living organism. For example, if an attacker uses a pre-recorded face video with an independent simulated finger mold, he can directly output a low liveness confidence score without entering the liveness classifier.

[0067] If the time phase difference is within the physiological synchronization time window, the time phase difference is used as a synchronization constraint factor. That is, the phase difference value is standardized and used as an additional feature dimension. It is then concatenated and fused with the spectral distribution feature vector and the mechanical creep feature vector, and input together into the pre-trained liveness classifier for comprehensive discrimination.

[0068] This linkage detection mechanism utilizes the cross-modal law of physiological synchronicity between the face and the finger on the time axis. Attackers find it difficult to accurately simulate the precise time phase relationship between the heartbeat vibration of the face and the pressing of the finger at the same time, thus effectively identifying the combined attack of face video playback and simulated finger mold.

[0069] Specifically, the pre-training process for a liveness classifier includes: Multiple sets of positive sample capacitance response timing signals of real living fingers under different ambient temperatures and different pressure levels were collected, as well as multiple sets of negative sample capacitance response timing signals of simulated fingers. The spectral distribution feature vector and mechanical creep feature vector of the positive sample capacitor response time-series signal and the negative sample capacitor response time-series signal are extracted and labeled to form a training dataset; The initial classifier model is trained in a supervised manner using the training dataset, and the model hyperparameters are optimized by cross-validation until the classification accuracy on the validation set reaches a preset standard, thus obtaining the pre-trained liveness classifier.

[0070] In this embodiment of the application, firstly, multiple sets of positive and negative sample data are collected in a laboratory environment.

[0071] Among them, positive sample collection uses the same model of capacitive fingerprint sensor as the deployment system to collect the entire process data of fingerprint pressing of real living people under different ambient temperatures and different pressing pressure conditions. Each person is collected more than 10 times under each condition to ensure that the positive sample covers the range of capacitive response changes of living fingers under various actual use scenarios.

[0072] Negative samples were collected using a variety of common counterfeit fingerprint materials, including at least silicone fingerprint films, conductive clay, 3D-printed resin materials, and multi-layered composite materials with an outer silicone layer encapsulating an inner conductive adhesive. For each counterfeit material, fingerprint pressure data was collected according to the same testing procedure.

[0073] Secondly, for each set of collected sample data, the spectral distribution feature vector and mechanical creep feature vector are extracted in the manner described above to form the original fusion feature vector. Each set of sample data is then labeled with a binary classification label, i.e., positive samples are labeled as live subjects and negative samples are labeled as non-live subjects. After the above operations, a training dataset containing a large number of labeled samples is constructed.

[0074] Next, the training dataset is divided into training, test, and validation sets proportionally, for example, in a 7:2:1 ratio. The initial classifier model is trained using supervised learning, for example, a lightweight neural network with two hidden layers, which has strong non-linear fitting capabilities.

[0075] During training, cross-validation is used, such as five-fold cross-validation, to optimize model hyperparameters, including learning rate, regularization coefficient, number of hidden layer neurons, etc. When the model's classification accuracy on the validation set reaches a preset standard, such as a classification accuracy of 98% or higher, training is stopped, the model parameters are saved, and a pre-trained live classifier is obtained.

[0076] For example, a lightweight neural network is used as the liveness classifier. This neural network is a fully connected network with two hidden layers. The dimension of the input layer is determined by the dimension of the fused feature vector, for example, 32 dimensions. The first hidden layer contains 64 neurons, and the second hidden layer contains 32 neurons. The output layer has a dimension of 1, and a Sigmoid activation function is used to output a probability value between 0 and 1 as the liveness confidence score. The hidden layers use the ReLU activation function to avoid the gradient vanishing problem, and Dropout regularization is introduced between the fully connected layers. The Dropout ratio is set to 0.3 to prevent overfitting.

[0077] During training, the Adam optimizer is used for parameter updates, with an initial learning rate of 1×10⁻³ and a batch size of 32 or 64. The loss function used is the binary cross-entropy loss function, suitable for binary classification tasks. An early stopping strategy is employed during training; training is terminated early when the validation set loss no longer decreases for several consecutive training epochs to prevent overfitting. The number of training epochs is determined based on the convergence of the validation set accuracy, typically set to 50 to 100 epochs.

[0078] S50: Based on environmental perception data, the face recognition confidence score, the fingerprint matching score, and the liveness confidence score are adaptively weighted and fused to generate a comprehensive security score. If the comprehensive security score is higher than a preset threshold, the unlocking action is performed; otherwise, the unlocking is refused and an alarm is triggered.

[0079] In this embodiment, the ambient light intensity and skin humidity of the fingers are collected, and the fusion weight coefficients corresponding to the face recognition confidence score, fingerprint matching score, and liveness confidence score are adjusted. The three scores are weighted and summed according to their respective weights to obtain a comprehensive security score. When the comprehensive security score is higher than the preset unlocking threshold, the smart lock is controlled to perform the unlocking action. When the comprehensive security score is lower than the threshold, it indicates that the user's identity verification has failed, and the unlocking request is rejected. At the same time, if the comprehensive security score is lower than the threshold multiple times in a row, a local alarm is triggered and an abnormal unlocking reminder is sent to the bound user terminal.

[0080] Specifically, based on environmental perception data, the face recognition confidence score, the fingerprint matching score, and the liveness confidence score are weighted and fused to generate a comprehensive security score, including: The current ambient light intensity and finger skin humidity are obtained, and the fusion weight coefficients corresponding to the face recognition confidence score, the fingerprint matching score and the liveness confidence score are dynamically adjusted according to the ambient light intensity and the finger skin humidity. The dynamic adjustment rule for the fusion weight coefficient is as follows: When the ambient light intensity is lower than the preset lower limit or higher than the preset upper limit, the weight coefficients of the liveness confidence score and the fingerprint matching score are increased, and the weight coefficient of the face recognition confidence score is decreased. When the skin moisture of the finger is lower than the preset lower limit or higher than the preset upper limit, the weight coefficients of the liveness confidence score and the face recognition confidence score are increased, and the weight coefficient of the fingerprint matching score is decreased. Each score is weighted and summed with its corresponding adjusted weight coefficient, and the weighted sum is normalized and then used as the comprehensive safety score output.

[0081] In this embodiment, the ambient light intensity and skin moisture status of the finger in the current unlocking scenario are obtained. The ambient light intensity is read by the existing ambient light sensor of the smart lock. This sensor is usually used in smart locks to automatically adjust the camera exposure parameters and panel screen brightness according to the ambient brightness. Reusing this sensor does not require additional hardware costs. If the smart lock is not equipped with an independent ambient light sensor, it can also be indirectly mapped through the automatic exposure control parameters of the camera. That is, when the automatic exposure gain value is higher than the preset gain threshold, it indicates that the ambient light is insufficient; when the exposure time is extremely short and the pixel value of the highlight area is saturated, it indicates that the ambient light is too strong.

[0082] Finger skin moisture is reflected based on the amplitude distribution of the aforementioned capacitance response timing signal and the average ridge width of the fingerprint image. Specifically, when the overall capacitance response amplitude is high and the average ridge width is wide, it indicates that the finger skin is moist, meaning that water fills the gaps between the ridges, causing the image to be blurry. When the overall capacitance response amplitude is low and the average ridge width is narrow, it indicates that the finger skin is dry and the signal is attenuated. Finger skin moisture is normalized to a range of 0 to 100, where 0 to 30 is the overly dry range, 30 to 70 is the normal range, and 70 to 100 is the overly moist range.

[0083] Secondly, based on the ambient light intensity and the skin moisture level of the fingers, the fusion weight coefficients corresponding to the three scores are dynamically adjusted. The dynamic adjustment strategy is as follows: Illumination compensation rule: When the ambient light intensity is lower than the preset lower limit threshold or higher than the preset upper limit threshold, for example, when the ambient light intensity is lower than 30 lux or higher than 20,000 lux, it indicates that the quality of the face image is greatly affected by the lighting conditions, and the credibility of the face recognition confidence score decreases. At this time, the weight coefficients of the liveness confidence score and the fingerprint matching score are increased, and the weight coefficient of the face recognition confidence score is decreased.

[0084] For example, the face confidence score has a weight of 0.25, the fingerprint matching score has a weight of 0.40, and the liveness confidence score has a weight of 0.35; when the lighting is abnormal, the weight is adjusted to 0.10 for the face confidence score, 0.45 for the fingerprint matching score, and 0.45 for the liveness confidence score.

[0085] Humidity compensation rule: When the skin humidity of the finger is lower than the preset lower humidity threshold or higher than the preset upper humidity threshold, for example, when the skin humidity of the finger is lower than 25 or higher than 75, it indicates that the skin condition of the finger is abnormal, which leads to a decrease in the quality of the fingerprint image and a decrease in the reliability of the fingerprint matching score. At this time, the weight coefficients of the liveness confidence score and the face recognition confidence score are increased, while the weight coefficient of the fingerprint matching score is decreased.

[0086] For example, when humidity is abnormal, the weights are adjusted to 0.35 for face confidence score, 0.15 for fingerprint matching score, and 0.50 for liveness confidence score.

[0087] Furthermore, ambient light intensity and finger skin moisture are assessed independently, triggering corresponding weight adjustments. That is, when both anomalies occur simultaneously, the weights for both face and fingerprint recognition decrease, while the weight for the liveness confidence score increases accordingly. This dynamic weight adjustment mechanism solves the problem of high false rejection rates in extreme environments caused by fixed-weighted fusion, enabling smart locks to maintain stable recognition performance even under non-ideal conditions such as insufficient light, excessive light, excessively dry fingers, and excessively wet fingers, thus enhancing the engineering practicality of the solution. For example, suppose a user is unlocking a smart lock. After completing the initial screening of facial identity, the fine verification of fingerprint identity, and the capacitive-mechanical dual-mode liveness detection, the user obtains a facial recognition confidence score of 0.72, a fingerprint matching score of 0.85, and a liveness confidence score of 0.91.

[0088] Among them, the face recognition confidence score was 0.72, indicating that the user's facial features have a high cosine similarity to a certain registered template in the local feature library, and the identity screening passed the initial screening; the fingerprint matching score was 0.85, indicating that the user's fingerprint details match the local fingerprint template well, and the identity confirmation is reliable; the liveness confidence score was 0.91, indicating that the liveness classifier's judgment that the current contact object is a real live finger is extremely reliable, far exceeding the classification threshold of non-live counterfeit materials.

[0089] Simultaneously, the ambient light intensity was acquired to be 2500 lux, which falls within the normal indoor lighting range. The preset normal lighting range is 30 lux to 20000 lux, and 2500 lux falls within this range. The finger skin humidity indicator value was 55, with a normalized range of 0 to 100. The preset normal humidity range is 25 to 75, and 55 falls within this range. Since both environmental parameters are within the normal range, no dynamic measurement compensation is required, and the preset baseline weight coefficients are used for fusion. That is, the face recognition confidence score has a weight of 0.25, the fingerprint matching score has a weight of 0.40, and the liveness confidence score has a weight of 0.35. The sum of the three weight coefficients is 1, which satisfies the normalization constraint.

[0090] Specifically, the weighting coefficients reflect the relative importance of each score in the overall security score. The fingerprint matching score has the highest weight because fingerprints are the most accurate basis for individual identity verification; the liveness confidence score is the second highest because liveness detection directly determines whether spoofing attacks are defended; and the face recognition confidence score has the lowest weight because faces are only used for initial identity screening, while the fine verification stage relies on fingerprints and liveness detection.

[0091] Furthermore, the overall security score = 0.72×0.25+0.85×0.40+0.91×0.35≈0.84. If the preset overall security threshold is 0.75, this preset overall security threshold setting takes into account both security requirements and user convenience. If the threshold is too high, legitimate users will be frequently and falsely rejected, while if the threshold is too low, attackers may be allowed to pass. The current overall security score of 0.84 is higher than 0.75, so the unlocking request is deemed legitimate, and the door opening action is executed.

[0092] In summary, compared with existing technologies, this application enhances the smart lock's defense against spoofing attacks by multimodal fusion of two biometric features, face and fingerprint, combined with a liveness detection mechanism based on capacitance response timing analysis. Furthermore, the dynamic weighted fusion strategy based on environmental awareness maintains stable recognition performance even in various non-ideal usage scenarios, balancing security and user experience. It addresses the pain points of traditional single-modal smart locks being vulnerable to attacks and experiencing decreased recognition accuracy in complex environments, demonstrating significant practical application value.

[0093] In summary, the embodiments of this application have at least the following technical effects: This application provides a smart lock unlocking method based on multimodal biometric fusion. First, building upon static identity feature matching using both face and fingerprint, a dynamic liveness detection mechanism based on capacitive response timing signals is introduced. By extracting the spectral distribution feature vector and mechanical creep feature vector during finger pressing, a genuine live finger can be effectively distinguished from a fake one. Simultaneously, by combining the facial micro-vibration periodic signal with the time phase difference at the start of finger pressing, a multimodal physiological synchronicity constraint is constructed, enhancing the defense against complex deception attacks such as simulated fakes. Furthermore, the fusion weights of each biometric score are dynamically adjusted based on ambient light intensity and finger skin humidity, achieving adaptive optimization of the recognition strategy in complex environments. This ensures high security while reducing false rejection rates, balancing user experience and unlocking reliability.

[0094] Through the above technical solution, this application solves the problems of existing smart locks, such as the susceptibility of single biometric recognition to counterfeiting and the lack of liveness detection capabilities, and the inability of fixed multimodal fusion methods to adapt to complex and changing usage environments. It can effectively resist various common deception attacks such as photos, videos, and fake fingerprints. Even in abnormal lighting environments such as low light and backlight, as well as poor fingerprint collection conditions such as excessively dry or wet fingers, it can still maintain a stable recognition accuracy rate. This not only improves the overall security of smart lock unlocking, but also avoids the problem of excessively high false rejection rate in complex environments, balancing security and ease of use, and is suitable for the promotion and application of smart locks in both home and commercial settings.

[0095] Example 2, as Figure 3As shown, based on the same inventive concept as the smart lock unlocking method based on multimodal biometric fusion provided in Embodiment 1, this application also provides a smart lock unlocking system based on multimodal biometric fusion, including: The face feature matching module 11 is used to collect user facial images, extract global facial structure feature vectors, match them with the local face feature library, and output a face recognition confidence score. The fingerprint feature matching module 12 is used to simultaneously collect the fingerprint image sequence and capacitive response timing signal during the user's finger pressing process if the face recognition confidence score is higher than the face confidence threshold, perform ridge feature matching on the fingerprint image sequence, and output the fingerprint matching score. The capacitance-mechanical data acquisition module 13 is used to extract the capacitance-time dynamic curve of the entire finger pressing process from the capacitance response time sequence signal in response to the fingerprint matching score exceeding the fingerprint confidence threshold, perform frequency domain transformation on the capacitance-time dynamic curve to obtain the spectral distribution feature vector, and extract the ridge deformation recovery rate of the finger pressing in the early stage from the fingerprint image sequence to obtain the mechanical creep feature vector. The liveness detection module 14 is used to input the spectral distribution feature vector and the mechanical creep feature vector into a pre-trained liveness classifier and output a liveness confidence score. The unlocking execution module 15 is used to adaptively weight and fuse the face recognition confidence score, the fingerprint matching score and the liveness confidence score based on environmental perception data to generate a comprehensive security score. If the comprehensive security score is higher than a preset threshold, the unlocking action is executed; otherwise, the unlocking is refused and an alarm is triggered.

[0096] In one embodiment, the face feature matching module 11 is specifically used for: Perform face detection and alignment on the facial image, and crop out the face region; The global structural feature vector of the face region is extracted using a pre-trained lightweight convolutional neural network; Calculate the cosine similarity between the global structural feature vector and each template feature vector in the local face feature library, and output the maximum similarity value as the face recognition confidence score; If the facial recognition confidence score is lower than the facial confidence threshold, the smart lock's fill light will be turned on and the facial image will be re-acquired. After a preset number of consecutive failures, the lock will enter a locked state.

[0097] Furthermore, in one embodiment, the fingerprint image sequence and capacitive response timing signal during the user's finger pressing process are simultaneously acquired, and the fingerprint image sequence is subjected to ridge feature matching to output a fingerprint matching score, including: Based on the capacitive fingerprint sensor, multiple frames of fingerprint images are continuously acquired from the process of finger contact to full pressing at a preset sampling frame rate to form the fingerprint image sequence, and the capacitance response value corresponding to each frame of fingerprint image is recorded synchronously to form the capacitance response timing signal. Select a frame of fingerprint image from the fingerprint image sequence at the moment of stable pressure, extract the minutiae features, and perform minutiae matching with the local fingerprint feature library. Output the matching score as the fingerprint matching degree score.

[0098] Further, in one embodiment, in response to the fingerprint matching score exceeding the fingerprint confidence threshold, the capacitance-time dynamic curve of the entire finger pressing process is extracted from the capacitance response timing signal, and the capacitance-time dynamic curve is subjected to frequency domain transformation to obtain a spectral distribution feature vector, including: The capacitor response timing signal is preprocessed by denoising and normalization to obtain the capacitor-time dynamic curve; A fast Fourier transform is performed on the capacitance-time dynamic curve to extract the spectral amplitude distribution in the frequency band from 0.5 Hz to 15 Hz. The spectral amplitude distribution is integrated according to a preset frequency band interval, and the integrated energy value of each frequency band interval is used to form the spectral distribution feature vector; The preset frequency band range includes at least a low-frequency band that reflects involuntary finger muscle tremors and a mid-frequency band that reflects tissue elastic resonance.

[0099] Further, the ridge deformation recovery rate at the initial stage of finger pressure is extracted from the fingerprint image sequence to obtain a mechanical creep feature vector, including: Extract the first N frames of fingerprint images from the fingerprint image sequence after the finger touches the surface of the smart lock, where N is a preset positive integer; Calculate the mean ridge width and ridge direction distribution entropy frame by frame for the first N frames of fingerprint images; The rate of change of the average ridge width over time is taken as the ridge deformation recovery rate. The mechanical creep feature vector is formed by combining the ridge deformation recovery rate and the trend of the ridge direction distribution entropy.

[0100] In one embodiment, the liveness detection module 14 is specifically used for: The spectral distribution feature vector and the mechanical creep feature vector are normalized and then concatenated to form a fused feature vector; The fused feature vector is input into a pre-trained lightweight neural network classifier, and the probability value output by the liveness classifier is used as the liveness confidence score. The training samples for the pre-trained liveness classifier include capacitive response data of real live fingers and capacitive response data of various non-liveness imitation materials.

[0101] Furthermore, before inputting the spectral distribution feature vector and the mechanical creep feature vector into the pre-trained liveness classifier, the method further includes: Obtain the pixel optical flow change sequence of the cheek region in the facial image, and extract the facial micro-vibration periodic signal; The finger pressing start time in the capacitive response timing signal is acquired synchronously, and the time phase difference between the facial micro-vibration period signal and the finger pressing start time is calculated. The time phase difference is used as a synchronization constraint factor, and it is concatenated and fused with the spectral distribution feature vector and the mechanical creep feature vector, and then input into the pre-trained liveness classifier. If the time phase difference exceeds the preset human physiological synchronization time window, it is directly determined that there is a mismatch.

[0102] Specifically, the pre-training process for the liveness classifier includes: Multiple sets of positive sample capacitance response timing signals of real living fingers under different ambient temperatures and different pressure levels were collected, as well as multiple sets of negative sample capacitance response timing signals of simulated fingers. The spectral distribution feature vector and mechanical creep feature vector of the positive sample capacitor response time-series signal and the negative sample capacitor response time-series signal are extracted and labeled to form a training dataset; The initial classifier model is trained in a supervised manner using the training dataset, and the model hyperparameters are optimized by cross-validation until the classification accuracy on the validation set reaches a preset standard, thus obtaining the pre-trained liveness classifier.

[0103] Furthermore, in one embodiment, based on environmental perception data, the face recognition confidence score, the fingerprint matching score, and the liveness confidence score are weighted and fused to generate a comprehensive security score, including: The current ambient light intensity and finger skin humidity are obtained, and the fusion weight coefficients corresponding to the face recognition confidence score, the fingerprint matching score and the liveness confidence score are dynamically adjusted according to the ambient light intensity and the finger skin humidity. The dynamic adjustment rule for the fusion weight coefficient is as follows: When the ambient light intensity is lower than the preset lower limit or higher than the preset upper limit, the weight coefficients of the liveness confidence score and the fingerprint matching score are increased, and the weight coefficient of the face recognition confidence score is decreased. When the skin moisture of the finger is lower than the preset lower limit or higher than the preset upper limit, the weight coefficients of the liveness confidence score and the face recognition confidence score are increased, and the weight coefficient of the fingerprint matching score is decreased. Each score is weighted and summed with its corresponding adjusted weight coefficient, and the weighted sum is normalized and then used as the comprehensive safety score output.

Claims

1. A smart lock unlocking method based on multimodal biometric fusion, characterized in that, The method includes: Collect user facial images, extract global facial structure feature vectors, match them with a local face feature database, and output a face recognition confidence score. If the face recognition confidence score is higher than the face confidence threshold, the fingerprint image sequence and capacitive response timing signal during the user's finger pressing process are collected simultaneously, the fingerprint image sequence is matched with ridge features, and the fingerprint matching score is output. In response to the fingerprint matching score exceeding the fingerprint confidence threshold, the capacitance-time dynamic curve of the entire finger pressing process is extracted from the capacitance response time sequence signal, the capacitance-time dynamic curve is transformed in the frequency domain to obtain the spectral distribution feature vector, and the ridge deformation recovery rate at the initial stage of finger pressing is extracted from the fingerprint image sequence to obtain the mechanical creep feature vector. The spectral distribution feature vector and the mechanical creep feature vector are input into a pre-trained liveness classifier, which outputs a liveness confidence score. Based on environmental perception data, the face recognition confidence score, the fingerprint matching score, and the liveness confidence score are adaptively weighted and fused to generate a comprehensive security score. If the comprehensive security score is higher than a preset threshold, the unlocking action is performed; otherwise, the unlocking is refused and an alarm is triggered.

2. The smart lock unlocking method based on multimodal biometric fusion according to claim 1, characterized in that, The system acquires user facial images, extracts global facial structure feature vectors, matches them against a local facial feature database, and outputs a facial recognition confidence score, including: Perform face detection and alignment on the facial image, and crop out the face region; The global structural feature vector of the face region is extracted using a pre-trained lightweight convolutional neural network; Calculate the cosine similarity between the global structural feature vector and each template feature vector in the local face feature library, and output the maximum similarity value as the face recognition confidence score; If the facial recognition confidence score is lower than the facial confidence threshold, the smart lock's fill light will be turned on and the facial image will be re-acquired. After a preset number of consecutive failures, the lock will enter a locked state.

3. The smart lock unlocking method based on multimodal biometric fusion according to claim 1, characterized in that, Simultaneously acquire fingerprint image sequences and capacitive response timing signals during the user's finger pressing process, perform ridge feature matching on the fingerprint image sequences, and output a fingerprint matching score, including: Based on the capacitive fingerprint sensor, multiple frames of fingerprint images are continuously acquired from the process of finger contact to full pressing at a preset sampling frame rate to form the fingerprint image sequence, and the capacitance response value corresponding to each frame of fingerprint image is recorded synchronously to form the capacitance response timing signal. Select a frame of fingerprint image from the fingerprint image sequence at the moment of stable pressure, extract the minutiae features, and perform minutiae matching with the local fingerprint feature library. Output the matching score as the fingerprint matching degree score.

4. The smart lock unlocking method based on multimodal biometric fusion according to claim 1, characterized in that, In response to the fingerprint matching score exceeding the fingerprint confidence threshold, the capacitance-time dynamic curve of the entire finger pressing process is extracted from the capacitance response time-series signal. The capacitance-time dynamic curve is then subjected to frequency domain transformation to obtain a spectral distribution feature vector, including: The capacitor response timing signal is preprocessed by denoising and normalization to obtain the capacitor-time dynamic curve; A fast Fourier transform is performed on the capacitance-time dynamic curve to extract the spectral amplitude distribution in the frequency band from 0.5 Hz to 15 Hz. The spectral amplitude distribution is integrated according to a preset frequency band interval, and the integrated energy value of each frequency band interval is used to form the spectral distribution feature vector; The preset frequency band range includes at least a low-frequency band that reflects involuntary finger muscle tremors and a mid-frequency band that reflects tissue elastic resonance.

5. The smart lock unlocking method based on multimodal biometric fusion according to claim 1, characterized in that, The mechanical creep feature vector is obtained by extracting the ridge deformation recovery rate at the initial stage of finger pressure from the fingerprint image sequence, including: Extract the first N frames of fingerprint images from the fingerprint image sequence after the finger touches the surface of the smart lock, where N is a preset positive integer; Calculate the mean ridge width and ridge direction distribution entropy frame by frame for the first N frames of fingerprint images; The rate of change of the average ridge width over time is taken as the ridge deformation recovery rate. The mechanical creep feature vector is formed by combining the ridge deformation recovery rate and the trend of the ridge direction distribution entropy.

6. The smart lock unlocking method based on multimodal biometric fusion according to claim 1, characterized in that, The spectral distribution feature vector and the mechanical creep feature vector are input into a pre-trained liveness classifier, which outputs a liveness confidence score, including: The spectral distribution feature vector and the mechanical creep feature vector are normalized and then concatenated to form a fused feature vector; The fused feature vector is input into a pre-trained lightweight neural network classifier, and the probability value output by the liveness classifier is used as the liveness confidence score. The training samples for the pre-trained liveness classifier include capacitive response data of real live fingers and capacitive response data of various non-liveness imitation materials.

7. The smart lock unlocking method based on multimodal biometric fusion according to claim 6, characterized in that, Before inputting the spectral distribution feature vector and the mechanical creep feature vector into the pre-trained liveness classifier, the method further includes: Obtain the pixel optical flow change sequence of the cheek region in the facial image, and extract the facial micro-vibration periodic signal; The finger pressing start time in the capacitive response timing signal is acquired synchronously, and the time phase difference between the facial micro-vibration period signal and the finger pressing start time is calculated. The time phase difference is used as a synchronization constraint factor, and it is concatenated and fused with the spectral distribution feature vector and the mechanical creep feature vector, and then input into the pre-trained liveness classifier. If the time phase difference exceeds the preset human physiological synchronization time window, it is directly determined that there is a mismatch.

8. The smart lock unlocking method based on multimodal biometric fusion according to claim 6, characterized in that, The pre-training process for a liveness classifier includes: Multiple sets of positive sample capacitance response timing signals of real living fingers under different ambient temperatures and different pressure levels were collected, as well as multiple sets of negative sample capacitance response timing signals of simulated fingers. The spectral distribution feature vector and mechanical creep feature vector of the positive sample capacitor response time-series signal and the negative sample capacitor response time-series signal are extracted and labeled to form a training dataset; The initial classifier model is trained in a supervised manner using the training dataset, and the model hyperparameters are optimized by cross-validation until the classification accuracy on the validation set reaches a preset standard, thus obtaining the pre-trained liveness classifier.

9. The smart lock unlocking method based on multimodal biometric fusion according to claim 1, characterized in that, Based on environmental perception data, the face recognition confidence score, the fingerprint matching score, and the liveness confidence score are weighted and fused to generate a comprehensive security score, including: The current ambient light intensity and finger skin humidity are obtained, and the fusion weight coefficients corresponding to the face recognition confidence score, the fingerprint matching score and the liveness confidence score are dynamically adjusted according to the ambient light intensity and the finger skin humidity. The dynamic adjustment rule for the fusion weight coefficient is as follows: When the ambient light intensity is lower than the preset lower limit or higher than the preset upper limit, the weight coefficients of the liveness confidence score and the fingerprint matching score are increased, and the weight coefficient of the face recognition confidence score is decreased. When the skin moisture of the finger is lower than the preset lower limit or higher than the preset upper limit, the weight coefficients of the liveness confidence score and the face recognition confidence score are increased, and the weight coefficient of the fingerprint matching score is decreased. Each score is weighted and summed with its corresponding adjusted weight coefficient, and the weighted sum is normalized and then used as the comprehensive safety score output.

10. A smart lock unlocking system based on multimodal biometric fusion, characterized in that, A smart lock unlocking method based on multimodal biometric fusion as described in any one of claims 1 to 9, comprising: The face feature matching module is used to collect user facial images, extract global facial structure feature vectors, match them with the local face feature database, and output a face recognition confidence score. The fingerprint feature matching module is used to simultaneously collect the fingerprint image sequence and capacitive response timing signal during the user's finger pressing process if the face recognition confidence score is higher than the face confidence threshold, perform ridge feature matching on the fingerprint image sequence, and output the fingerprint matching score. The capacitance-mechanical data acquisition module is used to extract the capacitance-time dynamic curve of the entire finger pressing process from the capacitance response time sequence signal in response to the fingerprint matching score exceeding the fingerprint confidence threshold, perform frequency domain transformation on the capacitance-time dynamic curve to obtain the spectral distribution feature vector, and extract the ridge deformation recovery rate at the initial stage of finger pressing from the fingerprint image sequence to obtain the mechanical creep feature vector. The liveness detection module is used to input the spectral distribution feature vector and the mechanical creep feature vector into a pre-trained liveness classifier and output a liveness confidence score. The unlocking execution module is used to adaptively weight and fuse the face recognition confidence score, the fingerprint matching score, and the liveness confidence score based on environmental perception data to generate a comprehensive security score. If the comprehensive security score is higher than a preset threshold, the unlocking action is executed; otherwise, the unlocking is refused and an alarm is triggered.

Images