A visitor access control method and system based on multimodal biometric recognition

By synchronously collecting and encoding multimodal biometrics, and combining timestamps and weighted fusion, the problem of high computational cost and inaccurate tailgating detection in high-frequency access scenarios for multimodal biometric recognition is solved, achieving efficient and accurate visitor identity verification.

CN123072097APending Publication Date: 2026-11-10ANHUI TEDDY INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611231358.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-14
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing multimodal biometric identification methods suffer from high computational complexity and slow response speed in high-frequency traffic scenarios. Furthermore, tailgating detection is susceptible to environmental factors and lacks consistency verification between collected events and identity verification events, leading to misjudgments or missed judgments.

Method used

By using plantar pressure signals as a unified trigger source, face images, palm vein images, and gait joints are simultaneously acquired to generate multimodal feature packages with unified timestamps. These packages are then uniformly time-series encoded and feature normalized. Combined with the quality level labels and weights of the feature descriptors, the optimal candidate identity and overall confidence level are generated. The consistency between the number of acquired events and identity verification events is statistically analyzed to determine tailgating and execute corresponding operations.

Benefits of technology

It effectively reduces the computational complexity of identity recognition, improves recognition efficiency and accuracy, reduces false alarm rate and false negative rate, and achieves dual verification of tailgating behavior.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123072097A_ABST
    Figure CN123072097A_ABST
Patent Text Reader

Abstract

This invention belongs to the field of biometric recognition technology. It discloses a visitor access control method and system based on multimodal biometric recognition. The method includes using the plantar pressure signal generated when a visitor enters a preset collection area as a unified trigger source to simultaneously collect facial images, palm vein images, and gait joints, generating a multimodal feature package with a unified timestamp; receiving the multimodal feature package, performing unified temporal encoding and feature normalization to generate feature descriptors; statistically analyzing the proportion of frames with abrupt changes in cosine similarity between feature vectors at adjacent time points in the feature descriptors; and assigning corresponding quality level labels to the feature descriptors; retrieving candidate identities from a registered feature library based on the feature descriptors and performing time-warping comparisons to obtain comparison scores for each modality; thus improving visitor access efficiency, identity recognition accuracy, and access security.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of biometric recognition technology, and more specifically, to a visitor access control method and system based on multimodal biometric recognition. Background Technology

[0002] With the increasing demands for visitor access security in smart parks, corporate office buildings, and key locations, visitor access control technologies based on multimodal biometric recognition, such as facial recognition, palm vein recognition, and gait recognition, are being widely applied. Compared to single-biometric recognition, multimodal biometric recognition can effectively improve the accuracy and security of identity verification, and therefore is gradually becoming an important technical means for visitor identity verification.

[0003] However, existing multimodal biometric identification methods typically require meticulous time-series comparisons between the visitor to be identified and all registered visitors in the registration feature database. As the number of registered visitors increases, the large amount of time-series matching calculations leads to increased computational load for identity verification and decreased response speed, making it difficult to meet the real-time requirements of high-frequency visitor access scenarios. Furthermore, existing visitor access control methods largely rely on infrared beam detection, video target tracking, access control status checks, or people counting to detect tailgating behavior. These methods are easily affected by factors such as human occlusion, multiple people passing through in parallel, changes in ambient lighting, and repeated data collection. They also lack a consistency verification mechanism between data collection events and identity verification events, making it impossible to effectively verify the collected behavior based on the identity verification results. This can easily lead to misjudgments or omissions of tailgating, affecting visitor access efficiency and security.

[0004] In view of this, the present invention proposes a visitor access control method based on multimodal biometric recognition to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a visitor access control method based on multimodal biometric recognition, comprising: S1. Using the plantar pressure signal generated when a visitor enters the preset collection area as a unified trigger source, the system simultaneously collects facial images, palm vein images, and gait joints to generate a multimodal feature package with a unified timestamp. S2. Receive multimodal feature packets, perform unified temporal coding and feature normalization, generate feature descriptors, count the proportion of frames with abrupt changes in cosine similarity between feature vectors at adjacent time points in the feature descriptors, and assign corresponding quality level labels to the feature descriptors. S3. Based on the feature descriptor, retrieve candidate identities from the registered feature library and perform time-warping comparison to obtain the comparison scores of each modality. Assign corresponding weights to each modality according to the quality level label and perform weighted fusion to generate the optimal candidate identity and comprehensive confidence. S4. Based on the optimal candidate identity, count the number of events collected and the number of identity verification events within the preset time window. When the number of events collected is greater than the number of identity verification events, it is determined that there is tailgating and a tailgating anomaly is generated. Based on the tailgating anomaly, and according to the comparison result of the comprehensive confidence level and the preset confidence threshold, perform the operation of direct release, release after supplementary verification, or refuse release.

[0006] Preferably, the method for generating a multimodal feature packet with a uniform timestamp includes: When a visitor enters the preset acquisition channel, the foot pressure signal output by each pressure sensing unit of the preset pressure sensor array is continuously monitored. When the pressure values ​​of consecutive adjacent pressure sensing units all exceed the preset pressure value threshold, a unified trigger signal is generated. The unified trigger signal is simultaneously sent to the face image acquisition unit, the palm vein near-infrared acquisition unit, and the gait posture acquisition unit, so that each acquisition unit starts to acquire the corresponding biometric features at the same trigger time. Using a unified trigger time as the time reference, corresponding timestamps are assigned to face images, palm vein images, and gait joints, and temporal correspondences between modal features are established according to the unified timestamps. Face images, palm vein images, gait joints, and corresponding timestamps are associated and encapsulated to generate a multimodal feature package corresponding to the same visitor.

[0007] Preferably, the method for generating feature descriptors includes: The unified timestamp, face image, palm vein image and gait key points in the multimodal feature package are analyzed. A unified time axis corresponding to each modality data is established based on the unified timestamp. The face feature vectors corresponding to each time step are extracted from the face image according to the unified time axis and the first temporal feature sequence is formed. The palm vein texture feature vectors corresponding to each time step are extracted from the palm vein image and expanded according to a unified time axis to form a second temporal feature sequence. The joint point feature vectors corresponding to each time step are extracted from the gait joint points to form a third temporal feature sequence. The first, second, and third time-series feature sequences are normalized to map each modal feature to a uniform numerical range. Then, according to a uniform time axis and a preset feature organization format, the normalized time-series feature sequences are used to generate corresponding feature descriptors.

[0008] Preferably, the method for assigning corresponding quality level labels to feature descriptors includes: Calculate the cosine similarity between feature vectors at adjacent time points in each feature descriptor, identify cosine similarity mutation frames based on the change magnitude of adjacent cosine similarity, count the number of cosine similarity mutation frames in each feature descriptor, and calculate the proportion of the number of cosine similarity mutation frames to the total number of frames in the corresponding feature descriptor to obtain the corresponding proportion of cosine similarity mutation frames. The proportion of cosine similarity mutation frames is compared with a preset mutation frame proportion threshold. When the proportion of cosine similarity mutation frames is less than or equal to the preset mutation frame proportion threshold, a second quality level label is assigned to the corresponding feature descriptor. When the proportion of cosine similarity mutation frames is greater than the preset mutation frame proportion threshold, a first quality level label is assigned to the corresponding feature descriptor.

[0009] Preferably, the method for obtaining the comparison scores of each modality item includes: Calculate the coarse-grained mean vector corresponding to the visitor to be identified based on the feature descriptor, and calculate the cosine similarity between the coarse-grained mean vector and the coarse-grained mean vector corresponding to each registered visitor in the preset registration feature library. A candidate identity set is established by selecting a preset number of registered visitors with the highest cosine similarity. When there are registered visitors in the candidate identity set whose cosine similarity is greater than or equal to a preset cosine similarity threshold, the candidate identity set is used as the object of time bending comparison. When there are no registered visitors in the candidate identity set whose cosine similarity is greater than or equal to the preset cosine similarity threshold, the cosine similarity between the visitor to be identified and each registered visitor in the registration feature library is recalculated, and a preset number of registered visitors with the highest cosine similarity are selected to form a new candidate identity set. Dynamic time-bending comparison is performed on the feature descriptors of each modality of the visitor to be identified and the feature descriptors of the corresponding modalities of the candidate identities to obtain the comparison distance for each modality, and the sub-comparison scores of the corresponding modalities are generated based on the comparison distance.

[0010] Preferably, the method for generating the optimal candidate identity and overall confidence level includes: The quality level labels corresponding to the face feature descriptor, palm vein feature descriptor and gait feature descriptor are read respectively; according to the quality level label corresponding to each modality, the corresponding weight is assigned. When the corresponding modality carries the second quality level label, the first weight is assigned to the corresponding modality. When the corresponding modality carries the first quality level label, the second weight is assigned to the corresponding modality. The first weight is greater than the second weight. The scores of each modality are weighted and fused according to their corresponding weights, and the weighted fusion results are normalized to generate the comprehensive confidence score for each candidate identity. The comprehensive confidence scores of each candidate identity are compared, and the candidate identity with the highest comprehensive confidence score is selected as the optimal candidate identity. The comprehensive confidence score of the optimal candidate identity is then output.

[0011] Preferably, the statistical method for the number of data collection events and the number of identity verification events includes: When a multimodal biometric feature acquisition is completed, a corresponding acquisition event is generated and the time of the acquisition event is recorded; when the optimal candidate identity and comprehensive confidence score are generated, a corresponding identity verification event is generated and the optimal candidate identity and the time of the identity verification event are recorded. Establish the association between the data collection event and the identity verification event based on the optimal candidate identity; establish a preset time window based on the current time, and count the number of data collection events and the number of identity verification events associated with the optimal candidate identity within the preset time window.

[0012] Preferably, the method for generating trailing anomaly identifiers includes: The number of collection events and the number of identity verification events within the preset time window are obtained respectively; the time interval between two adjacent collection events is obtained and compared with the preset glitch filtering time window. When the time interval is less than the preset glitch filtering time window and the previous collection event corresponds to the identity verification event, the next collection event is determined to be a glitch collection event and the glitch collection event is removed from the collection event count. The number of collected events after removing burr-related events is compared with the number of identity verification events. If the number of collected events is not less than two and the number of identity verification events is one, it is determined that there is a mismatch between the number of collected events and identity verification events, indicating that there is tailgating, and a corresponding tailgating anomaly flag is generated.

[0013] Preferably, the method for performing direct release, release after supplementary verification, or rejection of release includes: Check whether a tailing anomaly flag is generated. If a tailing anomaly flag is generated, then perform a rejection and release operation. If no tailing anomaly flag is generated, then obtain the overall confidence score and compare it with a preset first confidence score threshold and a preset second confidence score threshold, respectively. The preset first confidence score threshold is greater than the preset second confidence score threshold. When the overall confidence level is greater than or equal to the preset first confidence level threshold, the direct release operation is performed; when the overall confidence level is less than the preset first confidence level threshold but greater than or equal to the preset second confidence level threshold, a supplementary verification process is performed, and the release operation is performed after the supplementary verification is passed; when the overall confidence level is less than the preset second confidence level threshold, the release operation is rejected.

[0014] A visitor access control system based on multimodal biometric recognition includes: The feature acquisition module is used to simultaneously acquire face images, palm vein images and gait joints by using the plantar pressure signal generated when a visitor enters the preset acquisition area as a unified trigger source, and generating a multimodal feature package with a unified timestamp. The feature encoding module is used to receive multimodal feature packets, perform unified temporal encoding and feature normalization, generate feature descriptors, count the proportion of frames with abrupt changes in cosine similarity between feature vectors at adjacent time points in the feature descriptors, and assign corresponding quality level labels to the feature descriptors. The identity assessment module is used to retrieve candidate identities from the registered feature library based on feature descriptors and perform time-warping comparison to obtain comparison scores for each modality. Based on the quality level label, corresponding weights are assigned to each modality and weighted fusion is performed to generate the optimal candidate identity and overall confidence. The access control module is used to count the number of events collected and the number of identity verification events within a preset time window based on the best candidate identity. When the number of events collected is greater than the number of identity verification events, it is determined that there is tailgating and a tailgating anomaly is generated. Based on the tailgating anomaly, and according to the comparison result of the comprehensive confidence level and the preset confidence threshold, the module performs the operation of direct passage, passage after supplementary verification, or refusal to pass.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention utilizes a coarse-grained mean vector to rapidly narrow down the candidate range for dynamic time-warping comparison, avoiding the need for all objects in the registered feature library to participate in complex temporal matching calculations. This effectively reduces the computational complexity in the identity recognition process and improves the efficiency of multimodal identity verification. Simultaneously, through a dynamic update mechanism for the candidate identity set, the candidate range can be adjusted in real time according to the current visitor characteristics to be identified, avoiding the problem of repeated full-library comparisons due to mismatches in the initial candidate set. This improves the real-time performance, stability, and accuracy of identity recognition under registered feature libraries of different sizes.

[0016] By introducing a burr acquisition event filtering mechanism, repeated acquisition events caused by factors such as human body pausing, body swaying, or center of gravity adjustment of the pressure sensor array are effectively eliminated, avoiding repeated triggering from interfering with the tailgating detection results. At the same time, by establishing a quantity consistency judgment mechanism between acquisition events and identity verification events, and using the identity verification results as the constraint basis for the authenticity of acquisition events, dual verification of tailgating behavior is achieved, improving the accuracy and reliability of tailgating identification and reducing the false alarm rate and false negative rate. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a visitor access control method based on multimodal biometric recognition according to the present invention. Figure 2This is a schematic diagram of a visitor access control system based on multimodal biometric recognition according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0019] Please see Figure 1 As shown, this embodiment provides a visitor access control method based on multimodal biometric recognition, specifically including the following steps: S1. Using the plantar pressure signal generated when a visitor enters the preset collection area as a unified trigger source, the system simultaneously collects facial images, palm vein images, and gait joints to generate a multimodal feature package with a unified timestamp. S2. Receive multimodal feature packets, perform unified temporal coding and feature normalization, generate feature descriptors, count the proportion of frames with abrupt changes in cosine similarity between feature vectors at adjacent time points in the feature descriptors, and assign corresponding quality level labels to the feature descriptors. S3. Based on the feature descriptor, retrieve candidate identities from the registered feature library and perform time-warping comparison to obtain the comparison scores of each modality. Assign corresponding weights to each modality according to the quality level label and perform weighted fusion to generate the optimal candidate identity and comprehensive confidence. S4. Based on the optimal candidate identity, count the number of events collected and the number of identity verification events within the preset time window. When the number of events collected is greater than the number of identity verification events, it is determined that there is tailgating and a tailgating anomaly is generated. Based on the tailgating anomaly, and according to the comparison result of the comprehensive confidence level and the preset confidence threshold, perform the operation of direct release, release after supplementary verification, or refuse release.

[0020] Methods for generating multimodal feature packets with uniform timestamps include: When a visitor enters the preset acquisition channel, the foot pressure signal output by each pressure sensing unit of the preset pressure sensor array is continuously monitored. When the pressure values ​​of consecutive adjacent pressure sensing units all exceed the preset pressure value threshold, a unified trigger signal is generated. The unified trigger signal is simultaneously sent to the face image acquisition unit, the palm vein near-infrared acquisition unit, and the gait posture acquisition unit, so that each acquisition unit starts to acquire the corresponding biometric features at the same trigger time. In this embodiment, it should be noted that the preset collection channel is set at a necessary location before visitors enter the controlled area, such as the entrance of the access control gate, the entrance of the office building lobby, or the entrance of the laboratory. Its length is preferably set to a distance that can accommodate one visitor to walk naturally, such as 1.5m to 3.0m, and its width is preferably the same as the single-person passage width, such as 0.6m to 1.0m, so as to ensure that only one visitor is allowed in the main collection area at any time, thereby reducing the overlap of biometric features caused by multiple people entering at the same time.

[0021] The preset acquisition channel is equipped with a pressure sensor array, a face image acquisition unit, a palm vein near-infrared acquisition unit, and a gait posture acquisition unit. All acquisition devices are connected to a unified acquisition controller, which is responsible for unified triggering and unified time management.

[0022] For example, in an office building access control scenario, a pressure sensor array can be laid on the ground in front of the gate, a face image acquisition unit can be installed at a height of about 1.5m directly in front of the gate, a palm vein near-infrared acquisition unit can be installed on the side of the gate where visitors can naturally reach out their hands, and a gait posture acquisition unit can be installed on the top of the passage, thus forming a complete multimodal acquisition area.

[0023] In this embodiment, the pressure sensing array is formed by arranging multiple pressure sensing units in a matrix. The pressure sensing units can be piezoresistive pressure sensor units, capacitive pressure sensor units, or flexible thin-film pressure sensor units. Each pressure sensing unit corresponds to a fixed sensing area on the channel floor and is used to detect the magnitude of the foot pressure in that area.

[0024] It should be noted that the pressure sensing unit in this embodiment is not limited to a specific sensor type, as long as it can output the pressure value of the corresponding area. For example, a flexible pressure mat can be used, which is composed of several flexible pressure sensing units; or an industrial pressure floor can be used, with each floor unit serving as a pressure sensing unit.

[0025] The system continuously collects pressure values ​​output by each pressure sensing unit and updates the pressure matrix according to a preset sampling period. When the pressure values ​​output by multiple consecutive adjacent pressure sensing units all exceed the preset pressure threshold, it is determined that the visitor's foot has stably entered the preset acquisition channel, rather than just a momentary step or edge contact, thereby generating a unified trigger signal.

[0026] The adjacent pressure sensing units can be determined according to their vertical, horizontal, or diagonal proximity in the matrix. The preset pressure threshold is set based on the range of the pressure sensing unit. For example, when the pressure sensing unit's range is 0~1000N, the preset pressure threshold can be set to 80N~150N, ensuring stable triggering by normal footsteps while minor disturbances such as dust, debris, or small animals do not trigger the acquisition process. For instance, if at a certain moment four consecutive adjacent pressure sensing units in the matrix output 132N, 145N, 138N, and 141N respectively, all exceeding the preset pressure threshold of 100N, the unified acquisition controller immediately generates a unified trigger signal and activates all biometric acquisition devices.

[0027] In terms of face image acquisition, the face image acquisition unit can employ existing technologies such as industrial cameras, binocular cameras, RGB cameras, or RGB-D cameras. This embodiment preferably uses a binocular industrial camera, which can simultaneously acquire images from both left and right perspectives, obtaining not only facial texture information but also facial depth information, thereby improving the stability of subsequent identity recognition.

[0028] In terms of palm vein acquisition, the near-infrared acquisition unit can utilize existing palm vein recognition technology, including near-infrared light-emitting devices and near-infrared image sensors. After near-infrared light irradiates the palm tissue, a stable palm vein texture image can be formed because the hemoglobin in the veins absorbs near-infrared light.

[0029] For gait acquisition, millimeter-wave human perception radar, such as 24GHz or 60GHz millimeter-wave radar, is preferred for the gait posture acquisition unit. The millimeter-wave radar transmits millimeter-wave signals and receives reflected echoes from the human body. After range processing, Doppler processing, and angle estimation, it obtains human point cloud data. Then, it performs clustering and tracking on the human point clouds at consecutive time points to identify the same human target. Subsequently, based on the connection relationships of the human skeleton and the spatial distribution characteristics of various parts of the body, it performs skeleton fitting on the human point cloud, extracting the spatial coordinates of human joints such as the shoulders, hips, knees, and ankles. The coordinates of the joints at each time point are arranged according to the sampling time sequence, forming a gait joint point sequence reflecting the human movement process. For example, if the millimeter-wave radar continuously outputs human point clouds at a sampling frequency of 20Hz, it can obtain 20 frames of human joint point data arranged in chronological order.

[0030] Using a unified trigger time as the time reference, corresponding timestamps are assigned to face images, palm vein images, and gait joints, and temporal correspondences between modal features are established according to the unified timestamps. Face images, palm vein images, gait joints, and corresponding timestamps are associated and encapsulated to generate a multimodal feature package corresponding to the same visitor.

[0031] In this embodiment, each acquisition unit is connected to a unified acquisition controller and shares a unified system clock. When generating a unified trigger signal, the unified acquisition controller simultaneously records the corresponding system time as a unified start time, and each acquisition device uses this start time as the acquisition time reference. During the acquisition process, each acquisition device continuously outputs data according to its own sampling frequency, and the unified acquisition controller assigns a unified timestamp to each frame of acquired data. For example, if a face image is acquired at 30fps, the timestamp interval between adjacent images is approximately 33.3ms; if a millimeter-wave radar outputs human body joint points at 20Hz, the timestamp interval between adjacent joint point sequences is approximately 50ms; a palm vein image is typically acquired in only one frame, and its timestamp corresponds to the specific moment when the acquisition was completed.

[0032] The unified acquisition controller establishes a time correspondence between data from different modalities based on the unified timestamps corresponding to each data point, enabling data within the same time period or closest time position to be correlated. For example, a nearest-time matching method can be used to group data with a time difference less than a preset time tolerance into the same acquisition time, thus ensuring a consistent time reference between different modalities. For instance, the timestamp corresponding to the 15th frame of a face image is 0.500s, and the timestamp corresponding to the 10th group of key points output by the millimeter-wave radar is 0.502s. The time difference between the two is only 2ms, which is less than the preset time tolerance of 5ms. Therefore, a correspondence is established, and they are considered as data acquired at the same time.

[0033] This data collection task is assigned a unique collection identifier, and the face image sequence, palm vein image, gait key point sequence, and corresponding unified timestamp are associated and stored according to a preset data structure. This allows data from different modalities to be indexed by the unified collection identifier, while maintaining the time correspondence based on the unified timestamp, ultimately generating a multimodal feature package corresponding to the same visitor.

[0034] Methods for generating feature descriptors include: The unified timestamp, face image, palm vein image and gait key points in the multimodal feature package are analyzed. A unified time axis corresponding to each modality data is established based on the unified timestamp. The face feature vectors corresponding to each time step are extracted from the face image according to the unified time axis and the first temporal feature sequence is formed. It should be noted that this embodiment uses a unified timestamp as a common time reference to establish a unified time axis, and divides the unified time axis into several consecutive time slices, each time slice corresponding to a fixed time position. Subsequently, based on the timestamps of each modal data, the corresponding data is mapped to its respective time slice, thereby establishing the time correspondence between the various modal data. For example, if the unified acquisition time lasts for 1 second, it can be divided into 20 consecutive time slices of 50ms, so the unified time axis contains 20 time positions; the facial image sampling frequency is 30fps, which can be mapped to adjacent time slices among the 20 time positions; the millimeter-wave radar sampling frequency is 20Hz, so each frame of gait joint points corresponds to a time position; the palm vein image is acquired only once, and its timestamp corresponds to the time slice in which the acquisition is completed.

[0035] After establishing a unified timeline, feature extraction is performed frame-by-frame on the face image sequence in chronological order. Specifically, each frame of the face image first undergoes face detection, face alignment, and scale normalization. Then, feature vectors reflecting facial texture, contour, and local structural information are extracted, generating one face feature vector for each frame. Subsequently, based on the unified timeline, the face feature vectors corresponding to each time position are arranged in chronological order to form the first temporal feature sequence. For example, if the unified timeline contains 20 time positions, 20 sets of face feature vectors are obtained, and then arranged according to time position... , ... The first temporal feature sequence is formed by arranging them sequentially.

[0036] The palm vein texture feature vectors corresponding to each time step are extracted from the palm vein image and expanded according to a unified time axis to form a second temporal feature sequence. The joint point feature vectors corresponding to each time step are extracted from the gait joint points to form a third temporal feature sequence. For palm vein images, since only one palm vein image is typically acquired during a single passage, the first step is to perform grayscale enhancement, texture enhancement, and vein region extraction on the palm vein image to obtain a palm vein texture feature vector that reflects the distribution characteristics of the palm vein patterns. Subsequently, using a unified timeline as a temporal reference, the palm vein texture feature vector is mapped to various time positions on the unified timeline, constructing a corresponding second temporal feature sequence. This ensures that the palm vein features maintain a consistent temporal organization with facial and gait features. For example, if the unified timeline contains 20 time positions, the same palm vein texture feature vector is associated with each of the 20 time positions, thus forming a second temporal feature sequence of length 20.

[0037] For gait joint sequence, the coordinate data of human joints at each sampling time are first read and organized sequentially according to a preset joint arrangement order, such as shoulder, elbow, wrist, hip, knee, and ankle. The coordinates of all joints at the same sampling time are combined to form the joint feature vector for that time. Then, based on a unified time axis, the joint feature vectors corresponding to each time position are arranged in chronological order to form a third temporal feature sequence. For example, when the unified time axis contains 20 time positions, 20 sets of human joint feature vectors can be obtained respectively, and arranged sequentially according to the unified time position to form the third temporal feature sequence.

[0038] The first, second, and third time-series feature sequences are normalized to map each modal feature to a uniform numerical range. Then, according to a uniform time axis and a preset feature organization format, the normalized time-series feature sequences are used to generate corresponding feature descriptors.

[0039] Because facial features, palm vein features, and gait joint features differ in numerical range, feature dimension, and data units, direct similarity calculations can easily lead to numerical biases between different modalities. Therefore, this embodiment normalizes the first, second, and third temporal feature sequences respectively.

[0040] Specifically, the maximum and minimum values ​​of each feature dimension in each time-series feature sequence are statistically analyzed. Then, linear normalization mapping is performed based on the maximum and minimum values ​​corresponding to each feature dimension, mapping each feature dimension to a uniform numerical range, such as the interval 0 to 1. This eliminates the influence caused by different dimensions between different modalities. For example, a certain dimension of facial features has a value range of 20 to 180, which is mapped to 0 to 1 after normalization; a certain dimension of gait joint coordinates has a value range of -0.5 to 0.5, which is also mapped to 0 to 1 after normalization, ensuring that different modalities have a consistent numerical scale.

[0041] After normalization, each temporal feature sequence is organized according to a preset feature organization format, using a unified timeline as an index. The preset feature organization format includes time position identifiers and corresponding normalized feature vectors for each time position, ensuring that each time position on the unified timeline corresponds to a unique set of normalized feature vectors. These time positions are arranged chronologically to form the feature descriptor for the corresponding modality. For example, when the unified timeline contains 20 time positions, the face modality forms a face feature descriptor containing 20 sets of normalized face feature vectors, the palm vein modality forms a palm vein feature descriptor containing 20 sets of palm vein texture feature vectors, and the gait modality forms a gait feature descriptor containing 20 sets of joint feature vectors.

[0042] Methods for assigning corresponding quality level labels to feature descriptors include: Calculate the cosine similarity between feature vectors at adjacent time points in each feature descriptor, identify cosine similarity mutation frames based on the change magnitude of adjacent cosine similarity, count the number of cosine similarity mutation frames in each feature descriptor, and calculate the proportion of the number of cosine similarity mutation frames to the total number of frames in the corresponding feature descriptor to obtain the corresponding proportion of cosine similarity mutation frames. It should be noted that, to avoid being influenced by individual differences among visitors when judging solely based on a single cosine similarity value, this embodiment calculates the variation range between adjacent cosine similarities and compares this variation range with a preset variation range threshold. When the variation range between adjacent cosine similarities exceeds the preset variation range threshold, the feature vector corresponding to the next time step is determined to be a cosine similarity abrupt change frame; otherwise, it is determined to be a normal continuous frame. By analyzing the changing trend of cosine similarity, rather than a single similarity value, abnormal changes occurring during the acquisition process can be identified more accurately.

[0043] For example, if the cosine similarity sequence corresponding to a facial feature descriptor is 0.96, 0.95, 0.94, 0.68, and 0.66 respectively, then the variation ranges of adjacent cosine similarities are 0.01, 0.01, 0.26, and 0.02 respectively. If the preset variation range threshold is 0.20, then the third variation range of 0.26 exceeds the preset variation range threshold. Therefore, the feature vector corresponding to the next time step is determined to be a cosine similarity mutation frame, and the other time positions are determined to be normal continuous frames.

[0044] The proportion of cosine similarity mutation frames is compared with a preset mutation frame proportion threshold. When the proportion of cosine similarity mutation frames is less than or equal to the preset mutation frame proportion threshold, a second quality level label is assigned to the corresponding feature descriptor. When the proportion of cosine similarity mutation frames is greater than the preset mutation frame proportion threshold, a first quality level label is assigned to the corresponding feature descriptor.

[0045] For example, a facial feature descriptor contains 20 time points, and if two frames exhibit abrupt changes in cosine similarity, then the proportion of cosine similarity abrupt changes is 10%. Similarly, a gait feature descriptor contains 20 time points, and if six frames exhibit abrupt changes in cosine similarity, then the proportion of cosine similarity abrupt changes is 30%. If the preset threshold for the proportion of abrupt changes is set to 20%, then the facial feature descriptor receives a second quality level label, and the gait feature descriptor receives a first quality level label. In subsequent identity assessment, different fusion weights can be assigned to different modalities based on their quality level labels, allowing modalities with better temporal stability to have a greater weight in the overall identity determination.

[0046] Methods for obtaining the comparison scores of each modality include: Calculate the coarse-grained mean vector corresponding to the visitor to be identified based on the feature descriptor, and calculate the cosine similarity between the coarse-grained mean vector and the coarse-grained mean vector corresponding to each registered visitor in the preset registration feature library. In this embodiment, it should be noted that after the facial feature descriptor, palm vein feature descriptor and gait feature descriptor are constructed, the coarse-grained mean vector corresponding to the visitor to be identified is calculated based on each modality feature descriptor. Since each modality feature descriptor is organized according to a unified time axis, the feature mean of each temporal feature descriptor at all time positions can be statistically analyzed to obtain a mean vector that can reflect the overall feature distribution of the corresponding modality.

[0047] Specifically, for the first temporal feature sequence, the values ​​of the same feature dimension at all time positions are averaged to obtain the corresponding face mean feature vector; for the second temporal feature sequence, the palm vein mean feature vector is calculated in the same way; and for the third temporal feature sequence, the gait mean feature vector is calculated. Then, the face mean feature vector, palm vein mean feature vector, and gait mean feature vector are concatenated in a preset order to form a coarse-grained mean vector corresponding to the visitor to be identified. Since this mean vector only reflects the overall features of each modality and does not retain specific temporal details, it can be used for rapid candidate identity retrieval in the registration feature library.

[0048] For example, a facial feature descriptor contains 20 time locations, each corresponding to a 128-dimensional feature vector. The average value of each of the 128 feature dimensions at the 20 time locations is calculated to obtain a 128-dimensional facial mean feature vector. The palm vein feature descriptor and gait feature descriptor respectively obtain their corresponding mean feature vectors. Finally, they are concatenated in the order of face, palm vein, and gait to form a coarse-grained mean vector corresponding to the visitor to be identified.

[0049] It should be noted that the pre-defined registration feature library can be deployed on a local server or an identity authentication server to store the identity information and biometric data of personnel with legitimate access permissions. During the personnel registration phase, facial images, palm vein images, and gait keypoint data of the registrants are collected respectively. Following the same feature extraction, unified temporal coding, and feature normalization process as in the identity verification phase, corresponding facial feature descriptors, palm vein feature descriptors, and gait feature descriptors are generated. Furthermore, a coarse-grained mean vector is calculated based on each modality feature descriptor, and the identity number, each modality feature descriptor, and the coarse-grained mean vector are associated and stored to form the registration feature library.

[0050] A candidate identity set is established by selecting a preset number of registered visitors with the highest cosine similarity. When there are registered visitors in the candidate identity set whose cosine similarity is greater than or equal to a preset cosine similarity threshold, the candidate identity set is used as the object of time bending comparison. Candidate identity set: ;in, Represents the set of candidate identities; Represents the first in the set of candidate identities One registered visitor; This represents the index number of the registered visitor in the candidate identity set; This represents the coarse-grained mean vector calculated based on the feature descriptor of the visitor to be identified. Indicates the first in the registration feature library The coarse-grained mean vector corresponding to each registered visitor; Indicates the visitor to be identified and the first Cosine similarity between coarse-grained mean vectors of registered visitors; This indicates a preset cosine similarity threshold; Quantifiers indicating existence; Indicates the first Each registered visitor belongs to the set. ; Indicates that it belongs to the symbol; When there are no registered visitors in the candidate identity set whose cosine similarity is greater than or equal to the preset cosine similarity threshold, the cosine similarity between the visitor to be identified and each registered visitor in the registration feature library is recalculated, and a preset number of registered visitors with the highest cosine similarity are selected to form a new candidate identity set. New set of candidate identities: ;in, Represents the updated set of candidate identities; Indicates all containing From the candidate set of registered visitors, the candidate set with the largest sum of cosine similarities between each registered visitor and the visitor to be identified is selected as the updated candidate identity set; Represents any candidate subset in the registered feature library; Represents a set For registering feature library A subset of; Indicates the registration feature library; Represents a set The number of registered visitors included; This represents the preset number of registered visitors that constitute the candidate identity set; Represents a set Includes a preset quantity One registered visitor; The subset symbol represents a subset where all elements in the first set belong to the second set.

[0051] Dynamic time-bending comparison is performed on the feature descriptors of each modality of the visitor to be identified and the feature descriptors of the corresponding modalities of the candidate identities to obtain the comparison distance for each modality, and the sub-comparison scores of the corresponding modalities are generated based on the comparison distance.

[0052] Specifically, using the feature descriptor of a certain modality of the visitor to be identified as the first time series sequence, and the feature descriptor of the modality corresponding to the candidate identity as the second time series sequence, the distance between the feature vectors corresponding to each time position in the two time series sequences is calculated, and the corresponding distance matrices are established. Subsequently, a cumulative distance matrix is ​​constructed based on the distance matrices, and the Dynamic Time Warping (DTW) algorithm is used to determine the optimal alignment path with the minimum cumulative distance, thus establishing a time correspondence between the first and second time series sequences. Since the Dynamic Time Warping algorithm allows for local time axis scaling, it can compensate for subtle differences in visitor walking speed, movement rhythm, and collection time, thereby improving the matching accuracy between time series features of different lengths.

[0053] After obtaining the optimal alignment path, the feature distances between corresponding time positions along the optimal alignment path are calculated, and the comparison distance of the corresponding modality is calculated based on each feature distance. The same method is used to perform dynamic temporal bending comparisons of face feature descriptors, palm vein feature descriptors, and gait feature descriptors, respectively, obtaining the face comparison distance, palm vein comparison distance, and gait comparison distance. For example, the gait feature descriptor of the visitor to be identified contains 20 time positions, while the gait feature descriptor corresponding to the candidate identity contains 22 time positions. Due to a slight difference in walking speed between the two, the dynamic temporal bending algorithm adjusts the local time axis to establish an optimal alignment path between the two gait sequences, and calculates the corresponding gait comparison distance based on the optimal alignment path; for example, the gait comparison distance is 0.12.

[0054] A distance-score mapping relationship is established based on the comparison distances corresponding to each modality, such that the smaller the comparison distance, the higher the corresponding sub-item comparison score; conversely, the larger the comparison distance, the lower the corresponding sub-item comparison score. The distance-score mapping relationship can be implemented using a linear mapping method, i.e., a maximum and minimum comparison distance are pre-defined, mapping the comparison distances of each modality to a unified score range, such as 0 to 100 points. When the calculated comparison distance is close to the minimum comparison distance, the corresponding sub-item comparison score is close to 100 points; when the calculated comparison distance is close to the maximum comparison distance, the corresponding sub-item comparison score is close to 0 points.

[0055] For example, a facial modality matching distance of 0.08 corresponds to a facial sub-item matching score of 96; a palm vein modality matching distance of 0.11 corresponds to a palm vein sub-item matching score of 91; and a gait modality matching distance of 0.18 corresponds to a gait sub-item matching score of 82. The final facial sub-item matching scores, palm vein sub-item matching scores, and gait sub-item matching scores are obtained respectively.

[0056] Methods for generating optimal candidate identities and overall confidence scores include: The quality level labels corresponding to the face feature descriptor, palm vein feature descriptor and gait feature descriptor are read respectively; according to the quality level label corresponding to each modality, the corresponding weight is assigned. When the corresponding modality carries the second quality level label, the first weight is assigned to the corresponding modality. When the corresponding modality carries the first quality level label, the second weight is assigned to the corresponding modality. The first weight is greater than the second weight. The scores of each modality are weighted and fused according to their corresponding weights, and the weighted fusion results are normalized to generate the comprehensive confidence score for each candidate identity. The comprehensive confidence scores of each candidate identity are compared, and the candidate identity with the highest comprehensive confidence score is selected as the optimal candidate identity. The comprehensive confidence score of the optimal candidate identity is then output.

[0057] For example, if the candidate identity A for the visitor to be identified has a face feature comparison score of 95, a palm vein feature comparison score of 90, and a gait feature comparison score of 85, where the face feature descriptor and palm vein feature descriptor correspond to the second quality level label, and the gait feature descriptor corresponds to the first quality level label, then the face modality and palm vein modality are assigned the first weight, and the gait modality is assigned the second weight, where the first weight is greater than the second weight. Subsequently, the comparison scores of each modality are weighted and fused according to their corresponding weights, and the weighted fusion result is normalized, resulting in a comprehensive confidence score of 0.94 for candidate identity A. The comprehensive confidence scores for the other candidate identities are calculated in the same way; for example, the comprehensive confidence scores for candidate identities B and C are 0.89 and 0.82, respectively. Since candidate identity A has the highest comprehensive confidence score, it is determined as the optimal candidate identity, and its comprehensive confidence score of 0.94 is output.

[0058] The statistical methods for the number of data collection events and the number of identity verification events include: When a multimodal biometric feature acquisition is completed, a corresponding acquisition event is generated and the time of the acquisition event is recorded; when the optimal candidate identity and comprehensive confidence score are generated, a corresponding identity verification event is generated and the optimal candidate identity and the time of the identity verification event are recorded. In this embodiment, it should be noted that after completing a multimodal biometric acquisition, a corresponding acquisition event is generated. Specifically, a complete biometric acquisition process is considered complete when a face image, palm vein image, and gait joint point are simultaneously acquired and the corresponding multimodal feature package is generated. At this time, an acquisition event record is generated for this acquisition process, and a unique acquisition event number is assigned to this acquisition event. The acquisition event record includes the acquisition event number, acquisition completion time, acquisition channel number, and the storage index of the corresponding multimodal feature package. Among them, the acquisition completion time uses a timestamp generated by the system's unified clock, and the multimodal feature package storage index is used to locate the face image, palm vein image, and gait joint point data corresponding to this acquisition. After the acquisition event is generated, the acquisition event record is written to the acquisition event queue in the order of acquisition completion time.

[0059] For example, when a visitor passes through the No. 1 collection channel, the system completes a multimodal biometric feature collection at 10:20:15, generates collection event E001, records the collection completion time as 10:20:15, the collection channel number as No. 1 collection channel, the multimodal feature package index as P001, and writes collection event E001 into the collection event queue.

[0060] After candidate identity retrieval, dynamic temporal bending comparison, and comprehensive confidence score calculation are completed, and the candidate identity with the highest comprehensive confidence score is determined as the optimal candidate identity, an identity verification event record is generated for this identity verification. The identity verification event record includes the identity verification event number, the optimal candidate identity identifier, the comprehensive confidence score, the verification completion time, and the corresponding data collection event number. The identity verification event number uniquely identifies an identity verification process, the optimal candidate identity identifier identifies the verification result, the comprehensive confidence score indicates the reliability of the identity verification, and the verification completion time uses a timestamp generated by the system's unified clock.

[0061] To establish the correspondence between identity verification results and the data collection process, the data collection event number corresponding to the multimodal feature package involved in this identity recognition is written into the identity verification event record, thus establishing a correspondence between the identity verification event and the data collection event. For example, if identity recognition is completed using the multimodal feature package corresponding to data collection event E001, and the optimal candidate identity is determined to be visitor A with a comprehensive confidence level of 0.96, then identity verification event V001 is generated, and the optimal candidate identity is recorded as visitor A, the comprehensive confidence level is 0.96, and the verification completion time is 10:20:15. Simultaneously, the corresponding data collection event number E001 is also recorded.

[0062] Establish the association between the data collection event and the identity verification event based on the optimal candidate identity; establish a preset time window based on the current time, and count the number of data collection events and the number of identity verification events associated with the optimal candidate identity within the preset time window.

[0063] Using the best candidate identity in the identity verification event as the association identifier, the corresponding collection event number is read to establish the correspondence between the best candidate identity, the collection event, and the identity verification event. When the same best candidate identity is identified again, the newly added collection event and identity verification event are continued to be associated with the event record set corresponding to that best candidate identity, thereby forming an event sequence organized by identity, so as to subsequently count the number of collection events and identity verification events corresponding to the same visitor.

[0064] For example, if visitor A completes two identity verifications in the morning, the corresponding data collection events E001 and E005 and identity verification events V001 and V005 will be generated. The system will associate all of these events with the event record set corresponding to visitor A. The event record set corresponding to visitor B will be established independently and will not be included in the event statistics of visitor A.

[0065] A preset time window is established based on the current time. The current system time is obtained as the statistical endpoint, and the corresponding time is traced back according to the preset time window length to determine the statistical start time. The preset time window is formed from the statistical start time to the current time. The preset time window length can be preset according to the actual application scenario, such as 2 seconds, 3 seconds, or 5 seconds.

[0066] Methods for generating trailing anomaly markers include: The number of collection events and the number of identity verification events within the preset time window are obtained respectively; the time interval between two adjacent collection events is obtained and compared with the preset glitch filtering time window. When the time interval is less than the preset glitch filtering time window and the previous collection event corresponds to the identity verification event, the next collection event is determined to be a glitch collection event and the glitch collection event is removed from the collection event count. Burr Collection Incident: ;in, This indicates a glitch event flag. The value is 1 when the second acquisition event is determined to be a glitch acquisition event, and 0 otherwise. Indicates the time when the first data collection event occurred; Indicates the time when the second data collection event occurred; Indicates the time interval between two consecutive data collection events; Indicates the preset glitch filtering time window; This indicates the valid identity verification event signal corresponding to the first data collection event. The value is 1 when the identity verification of the first data collection event has been completed, and 0 otherwise. In this embodiment, it should be noted that the preset glitch filtering time window refers to the time threshold used to determine whether two adjacent acquisition events belong to repeated triggers during the same visitor passage. It can be preset based on the statistical results of the acquisition channel length, normal visitor passage speed, pressure sensor array sampling period, and the time interval between two adjacent pressure triggers under normal passage conditions. When the time interval between two adjacent acquisition events is less than the preset glitch filtering time window, the second acquisition event is determined to be a repeated trigger generated during the same passage and is not included in the event count. When the time interval is greater than or equal to the preset glitch filtering time window, the second acquisition event is determined to correspond to a new acquisition behavior and participates in subsequent tailgating judgment. For example, statistically, the time interval between repeated triggers by normal visitors is mostly less than 0.2s. The preset glitch filtering time window can be set to 0.3s. When the time interval between two acquisition events is 0.15s, the second acquisition event is determined to be a repeated trigger; when the time interval is 0.6s, the second acquisition event is determined to correspond to a new acquisition behavior.

[0067] The number of collected events after removing burr-related events is compared with the number of identity verification events. If the number of collected events is not less than two and the number of identity verification events is one, it is determined that there is a mismatch between the number of collected events and identity verification events, indicating that there is tailgating, and a corresponding tailgating anomaly flag is generated.

[0068] Tailing anomaly indicator: ;in, This indicates a tailing anomaly flag. The value is 1 when tailing behavior is detected, and 0 otherwise. Indicates time The acquisition event signal is set to 1 when a multimodal biometric acquisition is completed and a corresponding acquisition event is generated; otherwise, it is set to 0. Indicates time The identity verification event signal takes a value of 1 when an identity verification is completed and the optimal candidate identity and comprehensive confidence score are generated; otherwise, it takes a value of 0. Indicates the start time of the current statistics; This represents any moment within the statistical time window; Indicates the preset time window length; indicates the time window based on time. Starting point, length is The statistical time interval; Indicates the number of events collected within the preset time window; Indicates the number of identity verification events within a preset time window; This represents a logical AND operation, which generates a trailing exception flag only if both left and right conditions are true.

[0069] The methods for performing direct release, release after supplementary verification, or rejection of release include: Check whether a tailing anomaly flag is generated. If a tailing anomaly flag is generated, then perform a rejection and release operation. If no tailing anomaly flag is generated, then obtain the overall confidence score and compare it with a preset first confidence score threshold and a preset second confidence score threshold, respectively. The preset first confidence score threshold is greater than the preset second confidence score threshold. When the overall confidence level is greater than or equal to the preset first confidence level threshold, the direct release operation is performed; when the overall confidence level is less than the preset first confidence level threshold but greater than or equal to the preset second confidence level threshold, a supplementary verification process is performed, and the release operation is performed after the supplementary verification is passed; when the overall confidence level is less than the preset second confidence level threshold, the release operation is rejected.

[0070] For example, the system presets a first confidence threshold of 0.95 and a second confidence threshold of 0.80. When a visitor's overall confidence score is 0.97 and no tailing anomaly is generated, the system directly allows passage because the overall confidence score is greater than the first confidence threshold. When another visitor's overall confidence score is 0.88 and no tailing anomaly is generated, the system performs a supplementary verification process because the overall confidence score is between the first and second confidence thresholds. After the supplementary verification is passed, the system allows passage. Supplementary verification refers to the use of auxiliary identity authentication methods, independent of multimodal biometric recognition, to further confirm the visitor's identity when the overall confidence level does not meet the direct release criteria but meets the minimum credibility requirements. Auxiliary identity authentication methods can include any one or any combination of visitor appointment QR code verification, ID document verification, SMS verification code verification, mobile terminal confirmation, remote confirmation by the person being visited, or manual confirmation by management personnel. If the supplementary verification passes, the visitor is released; if the supplementary verification fails, the visitor is denied.

[0071] When a visitor's overall confidence level is 0.72 and no tailgating anomaly flag is generated, the system will refuse passage because the overall confidence level is less than the second confidence level threshold. When a visitor's overall confidence level is 0.98 but a tailgating anomaly flag has been generated, it means that the system has determined that there is tailgating behavior. At this time, the overall confidence level will no longer be used for classification, and the system will directly refuse passage.

[0072] The preset pressure threshold is set by staff based on historical data analysis results. This historical analysis process includes the system collecting multiple pressure values ​​and calculating their average value as a reference to obtain the preset pressure threshold. Similarly, the preset mutation frame proportion threshold, preset cosine similarity threshold, preset first confidence threshold, and preset second confidence threshold are also set and adjusted by staff according to the system's historical operating data and the specific application scenario requirements. Example

[0073] Please see Figure 2 As shown, for parts not described in detail in this embodiment, please refer to the description in Embodiment 1. A visitor access control system based on multimodal biometric recognition is provided, including: The feature acquisition module is used to simultaneously acquire face images, palm vein images and gait joints by using the plantar pressure signal generated when a visitor enters the preset acquisition area as a unified trigger source, and generating a multimodal feature package with a unified timestamp. The feature encoding module is used to receive multimodal feature packets, perform unified temporal encoding and feature normalization, generate feature descriptors, count the proportion of frames with abrupt changes in cosine similarity between feature vectors at adjacent time points in the feature descriptors, and assign corresponding quality level labels to the feature descriptors. The identity assessment module is used to retrieve candidate identities from the registered feature library based on feature descriptors and perform time-warping comparison to obtain comparison scores for each modality. Based on the quality level label, corresponding weights are assigned to each modality and weighted fusion is performed to generate the optimal candidate identity and overall confidence. The access control module is used to count the number of data collection events and the number of identity verification events within a preset time window based on the best candidate identity. When the number of data collection events exceeds the number of identity verification events, tailgating is detected, and a tailgating anomaly flag is generated. Based on the tailgating anomaly flag and a comparison between the overall confidence level and a preset confidence threshold, the module performs actions such as direct passage, passage after supplementary verification, or denial of passage. All modules are connected via wired and / or wireless means to achieve data transmission between modules. Example

[0074] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the operation mode of the visitor access control method based on multimodal biometric recognition provided above.

[0075] Since the electronic device described in this embodiment is the one used in implementing the visitor access control method and system based on multimodal biometric recognition in this application embodiment, those skilled in the art can understand the specific implementation and various variations of the electronic device in this embodiment based on the visitor access control method and system based on multimodal biometric recognition described in this application embodiment. Therefore, how the electronic device implements the method in this application embodiment will not be described in detail here. Any electronic device used by those skilled in the art in implementing the visitor access control method and system based on multimodal biometric recognition in this application embodiment falls within the scope of protection of this application.

[0076] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0077] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A visitor access control method based on multimodal biometric recognition, characterized in that, include: S1. Using the plantar pressure signal generated when a visitor enters the preset collection area as a unified trigger source, the system simultaneously collects facial images, palm vein images, and gait joints to generate a multimodal feature package with a unified timestamp. S2. Receive multimodal feature packets, perform unified temporal coding and feature normalization, generate feature descriptors, count the proportion of frames with abrupt changes in cosine similarity between feature vectors at adjacent time points in the feature descriptors, and assign corresponding quality level labels to the feature descriptors. S3. Based on the feature descriptor, retrieve candidate identities from the registered feature library and perform time-warping comparison to obtain the comparison scores of each modality. Assign corresponding weights to each modality according to the quality level label and perform weighted fusion to generate the optimal candidate identity and comprehensive confidence. S4. Based on the optimal candidate identity, count the number of events collected and the number of identity verification events within the preset time window. When the number of events collected is greater than the number of identity verification events, determine that there is tailgating and generate a tailgating anomaly flag. Based on the tailing anomaly flag and the comparison between the overall confidence level and the preset confidence threshold, the system will perform actions such as direct release, release after supplementary verification, or rejection.

2. The visitor access control method based on multimodal biometric recognition according to claim 1, characterized in that, The method for generating multimodal feature packets with a unified timestamp includes: When a visitor enters the preset acquisition channel, the foot pressure signal output by each pressure sensing unit of the preset pressure sensor array is continuously monitored. When the pressure values ​​of consecutive adjacent pressure sensing units all exceed the preset pressure value threshold, a unified trigger signal is generated. The unified trigger signal is simultaneously sent to the face image acquisition unit, the palm vein near-infrared acquisition unit, and the gait posture acquisition unit, so that each acquisition unit starts to acquire the corresponding biometric features at the same trigger time. Using a unified trigger time as the time reference, corresponding timestamps are assigned to face images, palm vein images, and gait joints, and temporal correspondences between modal features are established according to the unified timestamps. Face images, palm vein images, gait joints, and corresponding timestamps are associated and encapsulated to generate a multimodal feature package corresponding to the same visitor.

3. The visitor access control method based on multimodal biometric recognition according to claim 2, characterized in that, The method for generating feature descriptors includes: The unified timestamp, face image, palm vein image and gait key points in the multimodal feature package are analyzed. A unified time axis corresponding to each modality data is established based on the unified timestamp. The face feature vectors corresponding to each time step are extracted from the face image according to the unified time axis and the first temporal feature sequence is formed. The palm vein texture feature vectors corresponding to each time step are extracted from the palm vein image and expanded according to a unified time axis to form a second temporal feature sequence. The joint point feature vectors corresponding to each time step are extracted from the gait joint points to form a third temporal feature sequence. The first, second, and third time-series feature sequences are normalized to map each modal feature to a uniform numerical range. Then, according to a uniform time axis and a preset feature organization format, the normalized time-series feature sequences are used to generate corresponding feature descriptors.

4. The visitor access control method based on multimodal biometric recognition according to claim 3, characterized in that, The method for assigning corresponding quality level labels to feature descriptors includes: Calculate the cosine similarity between feature vectors at adjacent time points in each feature descriptor, identify cosine similarity mutation frames based on the change magnitude of adjacent cosine similarity, count the number of cosine similarity mutation frames in each feature descriptor, and calculate the proportion of the number of cosine similarity mutation frames to the total number of frames in the corresponding feature descriptor to obtain the corresponding proportion of cosine similarity mutation frames. The proportion of cosine similarity mutation frames is compared with a preset mutation frame proportion threshold. When the proportion of cosine similarity mutation frames is less than or equal to the preset mutation frame proportion threshold, a second quality level label is assigned to the corresponding feature descriptor. When the proportion of cosine similarity mutation frames is greater than the preset mutation frame proportion threshold, a first quality level label is assigned to the corresponding feature descriptor.

5. A visitor access control method based on multimodal biometric recognition according to claim 4, characterized in that, The method for obtaining the comparison scores of each modality component includes: Calculate the coarse-grained mean vector corresponding to the visitor to be identified based on the feature descriptor, and calculate the cosine similarity between the coarse-grained mean vector and the coarse-grained mean vector corresponding to each registered visitor in the preset registration feature library. A candidate identity set is established by selecting a preset number of registered visitors with the highest cosine similarity. When there are registered visitors in the candidate identity set whose cosine similarity is greater than or equal to a preset cosine similarity threshold, the candidate identity set is used as the object of time bending comparison. When there are no registered visitors in the candidate identity set whose cosine similarity is greater than or equal to the preset cosine similarity threshold, the cosine similarity between the visitor to be identified and each registered visitor in the registration feature library is recalculated, and a preset number of registered visitors with the highest cosine similarity are selected to form a new candidate identity set. Dynamic time-bending comparison is performed on the feature descriptors of each modality of the visitor to be identified and the feature descriptors of the corresponding modalities of the candidate identities to obtain the comparison distance for each modality, and the sub-comparison scores of the corresponding modalities are generated based on the comparison distance.

6. A visitor access control method based on multimodal biometric recognition according to claim 5, characterized in that, The method for generating the optimal candidate identity and overall confidence level includes: The quality level labels corresponding to the face feature descriptor, palm vein feature descriptor and gait feature descriptor are read respectively; according to the quality level label corresponding to each modality, the corresponding weight is assigned. When the corresponding modality carries the second quality level label, the first weight is assigned to the corresponding modality. When the corresponding modality carries the first quality level label, the second weight is assigned to the corresponding modality. The first weight is greater than the second weight. The scores of each modality are weighted and fused according to their corresponding weights, and the weighted fusion results are normalized to generate the comprehensive confidence score for each candidate identity. The comprehensive confidence scores of each candidate identity are compared, and the candidate identity with the highest comprehensive confidence score is selected as the optimal candidate identity. The comprehensive confidence score of the optimal candidate identity is then output.

7. A visitor access control method based on multimodal biometric recognition according to claim 6, characterized in that, The statistical methods for the number of data collection events and the number of identity verification events include: When a multimodal biometric feature acquisition is completed, a corresponding acquisition event is generated and the time of the acquisition event is recorded; when the optimal candidate identity and comprehensive confidence score are generated, a corresponding identity verification event is generated and the optimal candidate identity and the time of the identity verification event are recorded. Establish the association between the data collection event and the identity verification event based on the optimal candidate identity; establish a preset time window based on the current time, and count the number of data collection events and the number of identity verification events associated with the optimal candidate identity within the preset time window.

8. A visitor access control method based on multimodal biometric recognition according to claim 7, characterized in that, The method for generating trailing anomaly identifiers includes: The number of collection events and the number of identity verification events within the preset time window are obtained respectively; the time interval between two adjacent collection events is obtained and compared with the preset glitch filtering time window. When the time interval is less than the preset glitch filtering time window and the previous collection event corresponds to the identity verification event, the next collection event is determined to be a glitch collection event and the glitch collection event is removed from the collection event count. The number of collected events after removing burr-related events is compared with the number of identity verification events. If the number of collected events is not less than two and the number of identity verification events is one, it is determined that there is a mismatch between the number of collected events and identity verification events, indicating that there is tailgating, and a corresponding tailgating anomaly flag is generated.

9. A visitor access control method based on multimodal biometric recognition according to claim 8, characterized in that, The methods for performing direct release, release after supplementary verification, or rejection of release include: Check whether a tailing anomaly flag is generated. If a tailing anomaly flag is generated, then perform a rejection and release operation. If no tailing anomaly flag is generated, then obtain the overall confidence score and compare it with a preset first confidence score threshold and a preset second confidence score threshold, respectively. The preset first confidence score threshold is greater than the preset second confidence score threshold. When the overall confidence level is greater than or equal to the preset first confidence level threshold, the direct release operation is performed; when the overall confidence level is less than the preset first confidence level threshold but greater than or equal to the preset second confidence level threshold, a supplementary verification process is performed, and the release operation is performed after the supplementary verification is passed; when the overall confidence level is less than the preset second confidence level threshold, the release operation is rejected.

10. A visitor access control system based on multimodal biometric recognition, used to implement the visitor access control method based on multimodal biometric recognition as described in any one of claims 1 to 9, characterized in that, include: The feature acquisition module is used to simultaneously acquire face images, palm vein images and gait joints by using the plantar pressure signal generated when a visitor enters the preset acquisition area as a unified trigger source, and generating a multimodal feature package with a unified timestamp. The feature encoding module is used to receive multimodal feature packets, perform unified temporal encoding and feature normalization, generate feature descriptors, count the proportion of frames with abrupt changes in cosine similarity between feature vectors at adjacent time points in the feature descriptors, and assign corresponding quality level labels to the feature descriptors. The identity assessment module is used to retrieve candidate identities from the registered feature library based on feature descriptors and perform time-warping comparison to obtain comparison scores for each modality. Based on the quality level label, corresponding weights are assigned to each modality and weighted fusion is performed to generate the optimal candidate identity and overall confidence. The access control module is used to count the number of events collected and the number of events verified within a preset time window based on the best candidate identity. When the number of events collected is greater than the number of events verified, it is determined that there is tailgating and a tailgating anomaly is generated. Based on the tailing anomaly flag and the comparison between the overall confidence level and the preset confidence threshold, the system will perform actions such as direct release, release after supplementary verification, or rejection.