An authentication method, system, device, and medium that integrates multimodal biometrics.

Abstract

This invention relates to the field of identity verification technology, specifically an identity verification method, system, device, and medium integrating multimodal biometrics. The method includes acquiring user features of a target user and extracting corresponding user feature vectors based on these features. The user features include facial biometrics, dynamic signature trajectory features, employee ID features, and password features. Based on the confidence level of each user feature vector, a target weight is determined for each user feature. The target feature vector is obtained by weighting each user feature vector and its corresponding target weight. Finally, the identity information of the target user is verified based on the target feature vector. The beneficial effects of this invention are that it not only effectively improves the system's adaptability and anti-counterfeiting level in complex application scenarios but also ensures high security and high availability of identity verification.

Description

Technical Field

[0001] This invention relates to the field of identity verification technology, and in particular to an identity verification method, system, device and medium that integrates multimodal biometrics. Background Technology

[0002] With the rapid development of information technology, electronic signatures are increasingly widely used in many fields, especially in the intelligent management and control systems for work tickets and operation tickets in hydropower plants. The use of electronic signatures can improve work efficiency, simplify processes, and ensure the traceability of operations. Currently, electronic signature anti-counterfeiting technology has made some progress. Common methods include password-based verification, simple biometric recognition, and static analysis of signature images. These technologies can, to a certain extent, meet basic anti-counterfeiting requirements.

[0003] However, existing electronic signature anti-counterfeiting technologies still have many problems and shortcomings: On the one hand, single authentication methods, such as relying solely on passwords or simple biometric identification, are at high risk of being forged or impersonated. For example, passwords are easily forgotten, leaked, or cracked, while the accuracy and reliability of simple biometric identification decrease significantly when faced with complex environmental conditions or spoofing attacks. On the other hand, traditional static signature image analysis methods struggle to capture dynamic features during the signing process, making them susceptible to forged signatures and unable to effectively identify abnormal signing behavior. Furthermore, these methods often exhibit low adaptability and robustness when facing different environmental conditions (such as changes in lighting and equipment differences), failing to meet the stringent requirements of the power industry and other sectors for both the legal validity and operational security of electronic signatures. Therefore, a more advanced, reliable electronic signature anti-counterfeiting method with multiple protection mechanisms is needed to address these issues. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides an identity verification method that integrates multimodal biometric features, including obtaining user features of a target user and extracting corresponding user feature vectors based on the user features, wherein the user features include facial biometric features, dynamic signature trajectory features, employee ID features, and password features; Based on the confidence level of each user's feature vector, determine the target weight corresponding to each user feature; The target feature vector is obtained by weighting each user feature vector and the target weight corresponding to each user feature. Verify the target user's identity information based on the target feature vector.

[0005] As a preferred embodiment of the multimodal biometric authentication method of the present invention, the method includes: determining the target weight corresponding to each user feature based on the confidence level of each user feature vector, including: When the confidence level of the feature vector of the facial biometrics is lower than the first preset threshold, the initial weight corresponding to the facial biometrics is reduced and the initial weight corresponding to the dynamic signature trajectory feature is increased, while the initial weights of the employee ID feature and the password feature remain unchanged.

[0006] As a preferred embodiment of the multimodal biometric authentication method of the present invention, the method further includes: determining the target weight corresponding to each user feature based on the confidence level of each user feature vector; and further includes: When the feature vector of the dynamic signature trajectory feature is determined to be abnormal based on the target triple verification, the initial weight corresponding to the dynamic signature trajectory feature is reduced and the initial weight corresponding to the facial biometric feature is increased, while the initial weights of the employee ID feature and the password feature remain unchanged. The target triple verification includes three types of verification: dynamics, behavioral patterns and spatiotemporal features.

[0007] As a preferred embodiment of the multimodal biometric authentication method of the present invention, the method further includes: determining the target weight corresponding to each user feature based on the confidence level of each user feature vector; and further includes: When the confidence level of the feature vector of the facial biometric feature is lower than the first preset threshold and the feature vector of the dynamic signature trajectory feature is determined to be abnormal based on the target triple verification, the initial weights corresponding to the facial biometric feature and the dynamic signature trajectory feature are reduced, while the initial weights of the employee ID feature and the password feature are increased.

[0008] As a preferred embodiment of the multimodal biometric authentication method of the present invention, wherein: the identity information of the target user is verified based on the target feature vector, including, The similarity between the target feature vector and the preset feature vector is calculated to obtain the target calculated value; When the target calculated value is greater than the second preset threshold, the target user's identity information is confirmed to be valid. Alternatively, if the target calculated value is less than or equal to the second preset threshold, the target user's identity information is deemed invalid.

[0009] As a preferred embodiment of the identity verification method integrating multimodal biometrics of the present invention, the user features include facial biometrics; Obtain the user characteristics of the target user, and extract the corresponding user feature vector based on the user characteristics, including: Obtain facial image information of the target user; The improved ResNet34 network model is used to extract the corresponding user feature vectors from facial image information.

[0010] As a preferred embodiment of the authentication method integrating multimodal biometrics of the present invention, the dynamic signature trajectory features include temporal features, dynamic features, and spatial features; Extract the corresponding user feature vector based on user characteristics, including: The temporal features, dynamic features, and spatial features are fused to obtain the user feature vector. The temporal features include the x and y coordinates after alignment by the dynamic time warping algorithm. The dynamic features include pressure features, velocity features, and acceleration features. The spatial features include the signature range and the center of gravity position.

[0011] Secondly, the present invention provides an identity verification method that integrates multimodal biometric features, including: an extraction module, used to obtain user features of a target user and extract corresponding user feature vectors based on the user features, wherein the user features include facial biometric features, dynamic signature trajectory features, employee ID features, and password features; The determination module is used to determine the target weights corresponding to each user feature based on the confidence level of each user feature vector; The weighting module is used to weight each user's feature vector and the target weight corresponding to each user's feature to obtain the target feature vector; The verification module is used to verify the identity information of the target user based on the target feature vector.

[0012] Thirdly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0013] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described above.

[0014] Compared with existing technologies, the beneficial effects of this invention are as follows: Through a dynamic confidence-weight coupling mechanism, failure factors in real-world scenarios such as deteriorating lighting and abnormal signatures are directly quantified into calculable confidence levels. Then, the weights of corresponding biometric features are adjusted in real time and compensated with other modalities, so that forged faces or highly imitated signatures cannot simultaneously lower the overall similarity. Furthermore, in the case of double failure limits, the weight pool is automatically shifted to employee ID and password, which not only safeguards the power compliance red line of FAR < 0.002%, but also avoids excessive reliance on static information, truly achieving a highly reliable identity verification effect under all working conditions. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating an authentication method that integrates multimodal biometrics.

[0017] Figure 2 This is a schematic diagram of the fusion logic.

[0018] Figure 3 This is a flowchart for verification. Detailed Implementation

[0019] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0020] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides an authentication method that integrates multimodal biometrics, including: S100: Obtain the user characteristics of the target user and extract the corresponding user feature vector based on the user characteristics. The user characteristics include facial biometric features, dynamic signature trajectory features, employee ID features, and password features. S200: Determine the target weight corresponding to each user feature based on the confidence level of each user feature vector; S300: The target feature vector is obtained by weighting each user feature vector and the target weight corresponding to each user feature. S400: Verify the identity information of the target user based on the target feature vector.

[0021] It should be noted that in real-world applications, electronic signature anti-counterfeiting systems are subject to significant variations in lighting conditions, writing devices, and writing postures during user authentication. This leads to substantial fluctuations in the quality of single biometric features (such as face or signature) acquisition and recognition confidence. For example, insufficient lighting can result in incomplete facial feature extraction, while differences in writing platforms or brief abnormal user behavior can distort signature trajectory features. Furthermore, traditional static weight fusion mechanisms cannot adapt to these real-time changes. Assigning a fixed high weight to a feature with low confidence reduces the accuracy and robustness of the overall authentication system, making it difficult to simultaneously reduce false acceptance and false rejection rates. Additionally, while employee IDs and passwords are highly secure static credentials, they are easily stolen. Using them independently or in simple combinations is insufficient to effectively address the risk of impersonation.

[0022] Therefore, to address the aforementioned issues of feature quality fluctuations, rigid fusion mechanisms, and insufficient anti-counterfeiting strength, this method, through steps S100-S400, first collects multimodal features and converts them into a computable vector form, providing a foundation for fusion analysis; then, it dynamically allocates the weights of each feature based on real-time confidence levels, automatically enhancing the contribution of reliable features when the quality of some features declines, thus achieving adaptive weighted fusion; finally, through high-precision comparison of comprehensive feature vectors, it achieves accurate verification of user identity. This method not only effectively improves the system's adaptability and anti-counterfeiting level in complex application scenarios but also ensures high security and high availability of identity verification.

[0023] Example 2, refer to Figures 1-3 As an embodiment of the present invention, based on the above embodiment, an authentication method integrating multimodal biometrics is provided.

[0024] In this embodiment of the application, step S100 involves obtaining the user characteristics of the target user and extracting the corresponding user feature vector based on the user characteristics. The user characteristics include facial biometric features, dynamic signature trajectory features, employee ID features, and password features, and include the following steps A1-A3:

[0025] A1: Obtain the facial image information of the target user.

[0026] It is understandable that obtaining the facial image information of the target user can be achieved by capturing the user's facial image through a camera, and further, the user's facial image is in 112×112 RGB format.

[0027] The improved ResNet34 network model is used to extract the corresponding user feature vectors from facial image information.

[0028] It should be noted that the improved ResNet34 network model (34-layer residual network model) is mainly reflected in setting the stride of the last layer to 1, so as to retain richer spatial detail information by increasing the resolution of the output feature map. Furthermore, an SE attention module (compression and activation module) is added to the network. The SE attention module adaptively calibrates the channel feature response, making the network pay more attention to the feature channels with more information and suppress useless channels. That is, after the 112×112 RGB format face image information is input into the improved ResNet34 network model, the output is a 512-dimensional floating-point vector (initial face feature vector). It should be noted that this vector is the basis for representing the face feature in high-dimensional space.

[0029] It should be further noted that the 512-dimensional initial face feature vector output by the improved ResNet34 network model needs to be normalized before the target face feature vector F can be obtained. Specifically, L2 normalization is used to output a 512-dimensional unit vector F with a magnitude of 1.

[0030] It is important to emphasize that the training of the improved ResNet34 network model employs a dynamic boundary adjustment strategy. This involves embedding a lightweight sub-network (such as a fully connected layer + sigmoid activation function) to predict a difficulty value based on the target face feature vector F as input. The difficulty value ranges from 0 to 1; a value closer to 1 indicates a more difficult sample (e.g., blurry image, occlusion, exaggerated pose, or very similar to a confused sample). It's crucial to note that this difficulty value is calculated in real-time based on the current features, not using a pre-set label. Furthermore, the margin parameter in the ArcFace loss function is dynamically adjusted based on the calculated difficulty value. The specific calculation formula is as follows: margin=base_margin+α×difficulty;

[0031] In the formula: margin represents the target margin value; base_margin represents the basic margin value (e.g., 0.5), corresponding to the boundary of the easiest sample; α represents the adjustment coefficient (e.g., 0.3), used to control the strength of the difficulty's influence on the margin; difficulty represents the difficulty value.

[0032] Furthermore, the calculated margin value is substituted into the ArcFace loss function so that the model outputs a more discriminative and robust facial feature vector.

[0033] A2: Dynamic signature trajectory features include temporal features, dynamic features, and spatial features.

[0034] By performing feature fusion processing on temporal features, dynamic features, and spatial features, a user feature vector is obtained. The temporal features include the x and y coordinates obtained after alignment by the Dynamic Time Warping (DTW) algorithm. The dynamic features include pressure features, velocity features, and acceleration features. The spatial features include the signature range and the center of gravity position.

[0035] Understandably, the acquisition of dynamic signature trajectory features can be achieved using high-precision input devices (such as Wacom tablets) to capture the signer's dynamic handwriting information. Further devices record at a high sampling rate of 200Hz to ensure that subtle changes during the writing process are captured and provide 2048 levels of pressure sensitivity to accurately reflect the continuous changes in pen pressure, so as to output raw, timestamped handwriting sequence data, including the pen tip coordinates (x, y) and pressure value (P) at each moment.

[0036] It should be noted that in order to eliminate interference variables unrelated to the writer's identity and to provide standardized, high-quality data for subsequent feature extraction, some preprocessing steps are required. Specifically, this includes using Kalman filtering or low-pass filtering to eliminate high-frequency noise introduced by physiological hand tremors or device vibration, thereby smoothing the handwriting trajectory to preserve the true writing trend. Furthermore, on the one hand, by scaling the size of the signature trajectory to a standard coordinate system, individual differences caused by magnification or reduction during writing are eliminated. On the other hand, by normalizing the time axis of the signature, the influence of writing speed is eliminated, making the duration of different signatures comparable. Finally, the smoothed and standardized handwriting sequence data is output.

[0037] Furthermore, for the preprocessed data, multi-dimensional features that can characterize writing habits are extracted. Specifically, by using a dynamic time warping algorithm, subtle fluctuations in the speed of each signature by the same writer are eliminated, thereby accurately calculating the distance between sequences to output an aligned, length-normalized coordinate sequence (64 dimensions).

[0038] Since it is difficult for imitators to mimic (copy) the physical quantities that subconsciously reflect the writing force and rhythm of the target user, obtaining dynamic characteristics is very important for anti-counterfeiting. The specific calculation formulas for velocity V and acceleration A are as follows: V = Δs1 / Δt;

[0039] In the formula: V represents velocity; Δs1 represents the displacement of the pen tip within the time interval (Δt), which is decomposed into displacement components in the x and y directions; Δt represents the time interval between adjacent sampling points, which is determined by the sampling rate of the sensor.

[0040] Understandably, a higher speed value indicates fast writing, which is common in the cursive parts of a signature; a lower speed value indicates slow writing, which is common in pauses or turning points in a signature.

[0041] V=ds1 / dt; In the formula, V represents instantaneous velocity; s1 represents displacement (or position), which can be represented by coordinates (such as x(t), y(t)); t represents time; ds1 / dt represents the derivative of displacement s1 with respect to time t, i.e. instantaneous velocity.

[0042] It is understandable that the changes in the velocity of the handwriting in the x and y directions can be analyzed using Vx=dx / dt and Vy=dy / dt respectively, in order to understand the changes in the direction of the handwriting.

[0043] A = ΔV / Δt; In the formula: A represents acceleration; ΔV represents the change in velocity, which is also decomposed into acceleration components in the x and y directions; Δt represents the time interval between adjacent sampling points, consistent with the interval in velocity calculation.

[0044] Understandably, a higher acceleration value indicates that the pen tip changes direction or pressure rapidly, which is common at the turning points of a signature; a lower acceleration value indicates writing in a straight line at a constant speed, which is common in the smooth sections of a signature.

[0045] A = d²s₁ / dt²; In the formula: A represents instantaneous acceleration; s1 represents displacement (or position), which can be represented by coordinates (such as x(t), y(t)); t represents time; d²s1 / dt² represents the second derivative of displacement s1 with respect to time t, i.e. instantaneous acceleration.

[0046] It is understandable that the changes in acceleration of the handwriting in the x and y directions can be analyzed using Ax=d²x / dt² and Ay=d²y / dt² respectively, in order to understand the dynamic characteristics of the handwriting.

[0047] The pressure value P can be recorded by the pressure sensor during the signature process. It should be noted that the extraction of dynamic features is to form an 18-dimensional feature vector.

[0048] Furthermore, extracting spatial features aims to capture the writer's macroscopic layout habits, as these habits are often subconscious and difficult to imitate. For example, whether a signature is written in a wide, flat, or tall, thin style, and whether the center of gravity of the handwriting is to the left, right, or centered. The signature range refers to the aspect ratio of the circumscribed rectangle of the signature. The circumscribed rectangle is the smallest axis-aligned rectangle that completely encloses the entire signature trajectory (i.e., the sides of the rectangle are parallel to the coordinate axes). It can be understood that when the aspect ratio is greater than 1, the signature is horizontally expansive, belonging to the wide, flat type; when the aspect ratio is approximately 1, the signature is roughly square; and when the aspect ratio is less than 1, the signature is vertically expansive, belonging to the tall, thin type. The center of gravity position refers to the geometric center of the signature trajectory, which is the average position of all stroke points in the signature. It can be understood that when the center of gravity is to the left, it means the writer habitually starts writing from the left, and the strokes are heavier on the left side; when the center of gravity is to the upper, it means the main part of the signature is concentrated in the upper right. It should be noted that the extraction of spatial features aims to form a 4-dimensional spatial feature vector.

[0049] Finally, a Long Short-Term Memory (LSTM) network model is used as a fusion unit to integrate the above-mentioned temporal, dynamic, and spatial features. This involves combining multiple pieces of information at each time point t (including coordinates and corresponding dynamic information such as velocity, pressure, and acceleration) into a comprehensive feature point. The entire time series is then input into the LSTM network, which learns the temporal dependencies and dynamic patterns in the signature process. Finally, a fixed 100-dimensional target signature feature vector S is output.

[0050] A3: Process the employee ID and password features to obtain the user feature vector.

[0051] It should be noted that there are two ways to obtain the corresponding user feature vector by encoding employee ID features: The first is based on word embedding techniques in natural language processing (such as Word2Vec and GloVe), treating each employee ID as a word and the entire company's employee ID list as a corpus. An embedding model is trained through supervised learning so that employee IDs from similar departments, job levels, or projects are closer in the vector space, while unrelated employee ID vectors are farther apart, outputting a low-dimensional, continuous employee ID feature vector; The second is based on binary encoding using a Bloom filter, which maps the employee ID string to multiple bits of a fixed-length (e.g., 256-bit) binary vector using multiple hash functions, sets these bits to 1, and outputs a 256-dimensional binary target employee ID feature vector.

[0052] The method to obtain the corresponding user feature vector by processing the cryptographic features can be based on cryptographic hash expansion. That is, first use a high-strength, salted key derivation function (such as PBKDF2-HMAC-SHA256) to iteratively hash the password (e.g., 10,000 times) and output a fixed-length hash value (e.g., 256 bits). Then, introduce a fully connected neural network, take the high-dimensional hash value as input, compress (encode) it into a low-dimensional continuous vector, and output a low-dimensional, continuous target cryptographic feature vector PW.

[0053] In this embodiment of the application, step S200 determines the target weight corresponding to each user feature based on the confidence level of each user feature vector, including the following contents B1-B3:

[0054] B1: When the confidence level of the feature vector of the facial biometrics is lower than the first preset threshold, reduce the initial weight of the facial biometrics and increase the initial weight of the dynamic signature trajectory feature, while keeping the initial weights of the employee ID feature and password feature unchanged.

[0055] It is understandable that poor ambient lighting can lead to low confidence in the feature vectors of facial biometrics.

[0056] It should be noted that the first preset threshold ranges from 0.5 to 0.85 (excluding both extremes). Furthermore, to address the low confidence level of facial biometric feature vectors due to poor ambient lighting, the initial weight of the facial biometric feature is reduced while the initial weight of the dynamic signature trajectory feature is increased. This is partly to reduce the influence of the facial biometric feature vector on the final decision, and partly because poor lighting typically does not affect the signature operation; therefore, as compensation, the weight of the relatively more reliable dynamic signature feature needs to be increased. Furthermore, since the employee ID and password are static credentials, their reliability does not change with environmental variations, so their weights remain unchanged.

[0057] Specifically, the formula for calculating the confidence level is as follows: ;

[0058] In the formula, Indicates confidence level; Represents the Sigmoid function; The cosine similarity between the current facial biometric feature F and the reference template (the pre-stored corresponding facial biometric feature) is represented by comparing the current facial feature vector F with the reference template feature vector F. ref The calculated value is: s0 represents the similarity baseline threshold, such as 0.7; k is the slope parameter, used to control the steepness of the Sigmoid function, with an example value of 10.

[0059] It is understandable that when the similarity s is greater than or equal to the baseline threshold s0, the confidence level... A similarity close to 1 indicates high confidence; when the similarity s is less than s0, the confidence level is low. It decays exponentially, indicating low confidence.

[0060] Furthermore, the corresponding weight adjustment formula is: ;

[0061] In the formula: This represents the adjusted weights of the facial biometric features; α represents the original weights of the facial biometric features. This represents the attenuation coefficient, ranging from [0,1], such as 0.5; Indicates the confidence level.

[0062] Understandably, when the confidence level When the weight decreases, This is done proportionally, further reducing the adjustment value of the weight corresponding to the facial biometric features is equivalent to increasing the adjustment value of the weight of the dynamic signature trajectory features. It should be noted that when the confidence level of the facial biometric feature vector is not lower than the first preset threshold, the weight... It equals the original weight α.

[0063] B2: When the feature vector of the dynamic signature trajectory feature is determined to be abnormal based on the target triple verification, the initial weight corresponding to the dynamic signature trajectory feature is reduced and the initial weight corresponding to the facial biometric feature is increased, while the initial weights of the employee ID feature and the password feature remain unchanged. The target triple verification includes three types of verification: dynamics, behavioral patterns and spatiotemporal features.

[0064] Understandably, dynamic verification refers to analyzing whether physical quantities such as speed, acceleration, and pressure exceed the user's historical normal range; behavioral pattern verification refers to analyzing whether behavioral characteristics such as stroke order, pause habits, and turning angles are consistent; and spatiotemporal feature verification refers to analyzing whether the overall shape, layout (spatial features), and combination with the time series of the signature are abnormal.

[0065] It should be noted that the aforementioned normal range of user history, consistency of behavioral characteristics, and abnormality are all threshold concepts. That is, within a certain range, it is considered to be within the normal range, and the threshold is set according to actual needs, therefore it is not subject to limitation. For cases where only the feature vector of the dynamic signature trajectory feature is abnormal, due to the suspicious nature of the signature, its corresponding weight needs to be reduced to prevent forgery attacks. As compensation, the initial weight corresponding to the facial biometric feature will be increased. Similarly, since the employee ID and password are static credentials, their reliability does not change with environmental changes, therefore their weights remain unchanged.

[0066] Specifically, the calculation formula for anomaly detection is as follows: ;

[0067] In the formula: Anomalys represents the degree of anomaly in the signature trajectory; v and a are the velocity and acceleration of the current signature, respectively; and These are the average of the user's historical signature speed and acceleration, respectively. and These are the standard deviations of the user's historical signature speed and acceleration, respectively.

[0068] It should be noted that when Anomalys are greater than a threshold (e.g., 3), If the signature is not found to be valid, then the current signature is considered abnormal.

[0069] Furthermore, the corresponding weight adjustment formula is: ;

[0070] In the formula: This indicates the weight corresponding to the adjusted dynamic signature trajectory features; This represents the original weight corresponding to the dynamic signature trajectory features; This represents the attenuation coefficient, ranging from [0,1], such as 0.6; This indicates an indicator function, which is 1 when Anomalys is greater than the threshold, and 0 otherwise.

[0071] Understandably, decreasing the adjustment value of the dynamic signature trajectory feature weight is equivalent to increasing the adjustment value of the corresponding weight of the facial biometric feature. It should be noted that when the feature vector of the dynamic signature trajectory feature is determined to be normal based on the target triple verification, the weight... equal to the original weight .

[0072] B3: When the confidence level of the feature vector of the facial biometric feature is lower than the first preset threshold and the feature vector of the dynamic signature trajectory feature is determined to be abnormal based on the target triple verification, the initial weights corresponding to the facial biometric feature and the dynamic signature trajectory feature are reduced, while the initial weights of the employee ID feature and the password feature are increased.

[0073] It should be noted that when the face confidence level is lower than the first preset threshold and the signature trajectory is abnormal, its influence must be significantly reduced. That is, authentication should be performed by using the weights corresponding to the employee number and password, which are relatively stable and have higher security, and supplemented by subsequent strong verification (such as SMS OTP) to complete the final identity authentication of the user.

[0074] Specifically, the corresponding weight adjustment formula is as follows: ;

[0075] In the formula: and γ and δ represent the weights of the adjusted employee ID feature and password feature, respectively; γ and δ represent the weights of the original employee ID feature and password feature, respectively; η represents the enhancement coefficient, used to control the magnitude of the weight increase, for example, 0.3. and represents the confidence levels of facial features and signature features, respectively. The calculation method and same.

[0076] Understandably, when there is no need to adjust the weights... and At that time, weight Equal to the original weight γ, weight It equals the original weight δ.

[0077] It is important to emphasize that the weights of employee ID and password features should be set with an additional upper limit. Specifically, γ+δ should be less than or equal to 0.5. Although the weights of static features (employee ID and password features) should be increased, their weights should not be too large to avoid over-reliance on relatively static information such as employee ID and password (after all, their security is relatively lower than that of dynamic biometric features).

[0078] In this embodiment of the application, step S300 involves weighting each user feature vector according to the target weight corresponding to each user feature to obtain the target feature vector, which includes the following:

[0079] It should be noted that the original weights for each user feature are set according to actual needs, but the weights for biometric features (face and signature) should be higher than those for static knowledge credentials (employee ID and password). For example, α=0.35, β=0.35, γ=0.15, δ=0.15.

[0080] Specifically, the weighted calculation formula is as follows: ;

[0081] In the formula: C is the target feature vector; ,in , , , There are no negative numbers in it.

[0082] It is understandable that the weights in this calculation process refer to the adjusted target weights when the above-mentioned situations occur.

[0083] It is further important to understand that the calculated target feature vector is no longer a feature of any single modality, but a comprehensive feature that integrates information from multiple sources.

[0084] In this embodiment of the application, step S400, which verifies the identity information of the target user based on the target feature vector, includes the following steps D1-D2:

[0085] D1: Calculate the similarity between the target feature vector and the preset feature vector to obtain the target calculated value.

[0086] It is understandable that the preset feature vector It refers to the template vector that is calculated and saved in the database after collecting multiple biometric features and static knowledge credentials of the target user during the registration phase through the same process. It represents the baseline features of the target user's identity.

[0087] Specifically, the calculation formula is as follows: ;

[0088] In the formula: C represents the target calculated value; C represents the target feature vector; This represents a predefined feature vector; This indicates the calculation of the dot product of the two vectors, used to reflect the consistency of their directions.

[0089] Understandably, the target calculated value is somewhere between The scalar values ​​between these two values, after normalization, have a range of [value range missing]. The closer the value is to 1, the higher the similarity, meaning that the current user identity information is more consistent with the preset user identity information.

[0090] D2: When the target calculated value is greater than the second preset threshold, the target user's identity information is determined to pass; or when the target calculated value is less than or equal to the second preset threshold, the target user's identity information is determined to fail.

[0091] It should be noted that the setting of the second preset threshold is not arbitrary. Instead, it is determined by analyzing the similarity distribution of a large number of positive samples (multiple verifications by the same user) and negative samples (different users or spoofing attacks), combined with the security requirements of the application scenario (such as requiring a false acceptance rate (FAR) of <0.002%), selecting a balance point on the ROC curve, and optimizing it through cross-validation. The value range of the second preset threshold is greater than or equal to 0.85 and less than 1. It is important to emphasize that the second preset threshold must be higher than the first preset threshold in the system to ensure that the final identity authentication has a sufficiently high overall credibility.

[0092] It should be further noted that when the target user's identity information fails to pass verification, the system provides multi-level remedial measures. Specifically, after the initial verification failure, the system will not immediately reject the attempt but will allow the user to re-verify 1-2 times (to avoid non-malicious temporary interference, such as a user's slip of the hand during the initial signature, a momentary obstruction of the camera, a sudden change in lighting, or a one-time input error). If the retry still fails, the system will initiate a degradation strategy: downgrading to single-modal verification (using only facial recognition or only verifying the signature trajectory) or initiating manual review (meaning that all data from this verification (captured facial images, signature trajectory, environmental information, etc.) is packaged and submitted to the security administrator for manual review). For user accounts that frequently fail verification, the system will not simply block access, but will automatically analyze the reasons for the failure to distinguish between malicious attacks and legitimate changes in user behavior. For example, the user may have changed to a new signature device, become rusty due to long-term inactivity, or have facial features changed due to age. If the change is deemed legitimate, the system can adapt and optimize the template by updating the user's registration template with the latest successful biometric data, thus keeping the model up-to-date and reducing the false rejection rate for legitimate users in the future. If the above measures are ineffective, or the system determines that the risk is extremely high, a backup strong authentication mechanism, such as SMS OTP (one-time password) or hardware key (such as a USB key), will be activated for the final authentication of the user's identity.

[0093] To verify the accuracy and effectiveness of this method, experimental verification was conducted:

[0094] The verification datasets included a public dataset (SVC-2004) and a self-built dataset. The internationally recognized standard signature verification dataset contained 100 users, each with 20 genuine signatures and 20 forged signatures. The self-built dataset contained 500 users, each with 50 signatures, and took into account different devices and environmental noise interference. The experimental results are shown in Table 1. Table 1 Evaluation Results

[0095] Understandably, the false acceptance rate (FAR) refers to the probability that a system incorrectly accepts an impersonator (illegal user). The lower the value, the more secure the system and the stronger its anti-counterfeiting capabilities. The false rejection rate (FRR) refers to the probability that a system incorrectly rejects an legitimate user. The lower the value, the better the user experience and the higher the convenience. The equal error rate (EER) represents the error rate when FAR and FRR are equal. That is, the lower the EER value, the better the overall accuracy and balance of the system.

[0096] As shown in Table 1 above, compared with the false acceptance rate (FAR) of traditional DTW, this method reduces it by about 66 times, which means that the security of this method has achieved a qualitative leap. Compared with the false rejection rate (FRR) of traditional DTW, this method reduces it by about 6.5 times, which means that the possibility of legitimate users being falsely rejected has also been greatly reduced, and the user experience has been significantly improved. Compared with the equal error rate (EER) of traditional DTW, this method reduces it by about 15 times, which also proves that this method has achieved a good balance between security and convenience.

[0097] In summary, this method, through a dynamic confidence-weight coupling mechanism, directly quantifies failure factors in real-world scenarios such as deteriorating lighting and abnormal signatures into calculable confidence levels. It then adjusts the weights of corresponding biometric features in real time and compensates with other modalities, preventing forged faces or highly realistic signatures from simultaneously lowering the overall similarity. Furthermore, in cases of dual failure limits, it automatically shifts the weight pool to employee ID and password, maintaining the power compliance red line of FAR < 0.002% while avoiding excessive reliance on static information, truly achieving a highly reliable identity verification effect across all operating conditions.

[0098] Example 3 illustrates an illustrative scheme for an authentication method integrating multimodal biometrics. It should be noted that the technical solution of this authentication system integrating multimodal biometrics is based on the same concept as the aforementioned authentication method integrating multimodal biometrics. Details not described in detail in the technical solution of the authentication system integrating multimodal biometrics in this embodiment can be found in the description of the aforementioned authentication method integrating multimodal biometrics.

[0099] This embodiment also provides an identity verification system that integrates multimodal biometrics, including:

[0100] The extraction module is used to obtain the user characteristics of the target user and extract the corresponding user feature vector based on the user characteristics. The user characteristics include facial biometric features, dynamic signature trajectory features, employee ID features, and password features.

[0101] The determination module is used to determine the target weights corresponding to each user feature based on the confidence level of each user feature vector;

[0102] The weighting module is used to weight each user's feature vector and the target weight corresponding to each user's feature to obtain the target feature vector;

[0103] The verification module is used to verify the identity information of the target user based on the target feature vector.

[0104] This embodiment also provides an electronic device suitable for authentication using multimodal biometrics, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the authentication method for multimodal biometrics as proposed in the above embodiment.

[0105] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the authentication method for fusing multimodal biometrics as proposed in the above embodiments.

[0106] The storage medium proposed in this embodiment and the authentication method for integrating multimodal biometrics proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0107] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0108] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An authentication method integrating multimodal biometrics, characterized in that: include, The user features of the target user are obtained, and the corresponding user feature vector is extracted based on the user features. The user features include facial biometric features, dynamic signature trajectory features, employee ID features, and password features. Based on the confidence level of each user feature vector, the target weight corresponding to each user feature is determined; The target feature vector is obtained by weighting each user feature vector and the target weight corresponding to each user feature. The identity information of the target user is verified based on the target feature vector.

2. The authentication method integrating multimodal biometrics as described in claim 1, characterized in that: The step of determining the target weight corresponding to each user feature based on the confidence level of each user feature vector includes: When the confidence level of the feature vector of the facial biometric feature is lower than the first preset threshold, the initial weight corresponding to the facial biometric feature is reduced and the initial weight corresponding to the dynamic signature trajectory feature is increased, while the initial weights of the employee ID feature and the password feature remain unchanged.

3. The authentication method integrating multimodal biometrics as described in claim 1, characterized in that: The step of determining the target weight corresponding to each user feature based on the confidence level of each user feature vector further includes: When the feature vector of the dynamic signature trajectory feature is determined to be abnormal based on the target triple verification, the initial weight corresponding to the dynamic signature trajectory feature is reduced and the initial weight corresponding to the facial biometric feature is increased, while the initial weights of the employee ID feature and the password feature remain unchanged. The target triple verification includes three types of verification: dynamics, behavioral patterns, and spatiotemporal features.

4. The authentication method integrating multimodal biometrics as described in claim 1, characterized in that: The step of determining the target weight corresponding to each user feature based on the confidence level of each user feature vector further includes: When the confidence level of the feature vector of the facial biometric feature is lower than the first preset threshold and the feature vector of the dynamic signature trajectory feature is determined to be abnormal based on the target triple verification, the initial weights corresponding to the facial biometric feature and the dynamic signature trajectory feature are reduced, and the initial weights of the employee ID feature and the password feature are increased.

5. The authentication method integrating multimodal biometrics as described in claim 4, characterized in that: The step of verifying the identity information of the target user based on the target feature vector includes: The similarity between the target feature vector and the preset feature vector is calculated to obtain the target calculated value; When the target calculated value is greater than the second preset threshold, it is determined that the target user's identity information has been verified. Alternatively, if the target calculated value is less than or equal to the second preset threshold, the target user's identity information is determined to be invalid.

6. The authentication method integrating multimodal biometrics as described in claim 5, characterized in that: The user characteristics include the facial biometric features; Obtaining the user characteristics of the target user, and extracting the corresponding user feature vector based on the user characteristics, including: Obtain facial image information of the target user; The user feature vector is extracted from the face image information using an improved ResNet34 network model.

7. The authentication method integrating multimodal biometrics as described in claim 6, characterized in that: The dynamic signature trajectory features include temporal features, dynamic features, and spatial features; The step of extracting the corresponding user feature vector based on the user features includes: The temporal features, dynamic features, and spatial features are fused to obtain the user feature vector. The temporal features include x and y coordinates obtained after alignment by a dynamic time warping algorithm. The dynamic features include pressure features, velocity features, and acceleration features. The spatial features include signature range and center of gravity position.

8. An authentication system integrating multimodal biometrics, employing the method described in any one of claims 1-7, characterized in that, include: The extraction module is used to obtain the user features of the target user and extract the corresponding user feature vector based on the user features, wherein the user features include facial biometric features, dynamic signature trajectory features, employee ID features, and password features; The determination module is used to determine the target weight corresponding to each user feature based on the confidence level of each user feature vector; The weighting module is used to weight each user feature vector and the target weight corresponding to each user feature to obtain the target feature vector; The verification module is used to verify the identity information of the target user based on the target feature vector.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images