Privacy-Preserving Open-Set Face Recognition Method Based on LoRa and Uncertainty Estimation

Abstract

This invention relates to the fields of information security and artificial intelligence technology, specifically to a privacy-preserving open-set face recognition method based on LoRa and uncertainty estimation, comprising the following steps: generating a hash code set; pre-training a basic feature encoder to extract facial features; mapping facial features to binary hash codes in the hash code set, defining them as a target encryption key; and constructing a key space; selecting a query sample; fine-tuning the basic feature encoder multiple times to a personalized feature encoder; converging the facial features of the query sample under different conditions to the mapped target encryption key; during fine-tuning of the basic feature encoder, jointly learning evidence values ​​about the output with the total loss; in the authentication stage, calculating uncertainty; if the uncertainty value is greater than an uncertainty threshold, rejecting the access request; if the uncertainty value is not greater than a pre-uncertainty threshold, performing a bit-by-bit matching between the predicted key and the encryption key in the database.

Description

Technical Field

[0001] This invention relates to the fields of information security and artificial intelligence technology, and more specifically, to a privacy-preserving open-set face recognition method based on LoRa and uncertainty estimation. Background Technology

[0002] With the widespread application of facial recognition (FR) technology in finance, security, smart homes, healthcare, and many other fields, the sensitive personal information contained in the facial images and feature data it relies on has raised growing privacy concerns. Facial data can not only directly identify an individual, but may also reveal visually identifiable sensitive attributes, including but not limited to gender, age, race, and health status. Therefore, how to effectively protect user privacy while fully utilizing the convenience of facial recognition has become a key challenge in the development and deployment of FR systems. Existing privacy-preserving facial recognition methods can be mainly categorized into three types: visual-removal methods, feature-transform methods, and biometric encryption methods.

[0003] (1) Visual Removal Methods: Principle: These methods aim to eliminate or mask visually sensitive information in facial images through image processing techniques such as anonymization, blurring, removal or replacement of key facial regions. For example, by decoupling visually relevant and identity-related components and training a model on the “cleaned” facial data.

[0004] Advantages: It can intuitively reduce the risk of privacy leaks of images and make it difficult for unauthorized personnel to visually identify individuals.

[0005] Disadvantages: ① Insufficient template security: Although some visual information is removed, facial templates generated by these methods often retain rich identity information. If these templates are obtained by malicious attackers during storage or transmission, they may still face the risk of reconstruction attacks. Attackers can use advanced generative models to reverse engineer the original face image from the protected template, leading to privacy breaches. ② High training costs: Most visual removal methods require retraining the FR model to adapt to processing processed images with less sensitive information. This is not only computationally expensive but may also be difficult to apply flexibly in practical deployments. ③ Incomplete protection: Simply eliminating visual cues does not completely solve the problem of sensitive attribute inference, nor does it provide encryption-level protection for the final generated feature template.

[0006] (2) Feature transformation methods: Principle: These methods focus on irreversible transformations (such as hashing, encryption, or random projection) of feature vectors extracted from face images to generate secure biometric templates. Examples include BioHash, MLPHash, and WiFaKey.

[0007] Advantages: Designed to provide stronger security guarantees, once the conversion is complete, the original features are difficult to recover from the converted template, thereby enhancing privacy protection.

[0008] Disadvantages: ① Performance degradation: To achieve irreversible transformation, these methods often introduce noise or information loss during the transformation process, which may lead to a decrease in the accuracy of face recognition. There is an inherent trade-off between privacy protection and recognition performance. ② Risk of the distance-preserving property: Many feature transformation methods strive to maintain the consistency of the relative distance between the original feature space and the transformed feature space in order to maintain recognition performance. That is, if the original feature A is closer to C than B, then the transformed A' will also be closer to C' than B'. While this "distance-preserving" property is beneficial for recognition, it becomes a potential weakness for reconstruction attacks. Attackers can exploit this property to iteratively optimize and gradually approach the original face, thereby bypassing privacy protection mechanisms.

[0009] (3) Biometric Encryption Methods: Principle: These methods aim to combine the fuzziness of biometrics with the precision of cryptographic keys. Through various techniques (such as fuzzy extractors, fuzzy vaults, key binding, etc.), an encryption key is generated or securely released from biometric data. These methods typically do not directly transform the original features for matching, but rather use biometrics as credentials for generating or unlocking encryption keys, thereby achieving authentication.

[0010] Advantages: ① It combines biometric authentication capabilities with the strict security of cryptography, tightly integrating the authentication process with encryption principles (such as data encryption and digital signatures). ② If key generation is successful, the generated key can be directly used for cryptographic applications such as encryption and signing, providing a higher level of security. ③ Theoretically, the revocability of biometric credentials can be achieved through key revocation and updating.

[0011] Disadvantages: ① Closed-set limitations: Existing hash-based biometric methods are typically designed for "closed-set" scenarios, meaning the system only identifies registered users and cannot effectively handle or reject unregistered "unknown" identities. In real-world open environments, FR systems need the ability to identify known users and reject unknown users. ② Reliance on Error Correction Codes (ECCs) and their vulnerabilities: Biometric data inherently possesses "fuzziness," meaning subtle differences exist between different samples of the same person. Cryptography, however, requires precise "one-to-one" matching. To bridge this variability, many existing biometric encryption systems rely on ECCs to correct small errors in the matching process. However, this method introduces new vulnerabilities, such as attackers potentially exploiting the error correction codebook through analysis or manipulation. Furthermore, fixed error boundaries limit its flexibility in highly variable open-set environments. ③ Key generation challenges: Generating stable and unique encryption keys from high-dimensional, continuous, and inherently variable biometric data while simultaneously satisfying the cryptographic requirements of irreversibility, determinism, and collision resistance is a significant technical challenge.

[0012] In summary, existing privacy-preserving face recognition methods still have significant limitations in terms of privacy protection strength, recognition performance, open set adaptability, and attack resistance. Visual removal methods are insufficient in terms of template security; feature transformation methods face performance degradation, reconstruction risks due to distance preservation characteristics, closed set limitations, and dependence on vulnerable ECC; while biometric encryption methods struggle with key stability, auxiliary data privacy risks, and the trade-off between performance and security. Therefore, there is an urgent need for a novel method that can achieve a better balance between strong privacy protection, high recognition accuracy, and open set adaptability, and effectively resist known and potential attacks to meet the growing market demand and stringent regulatory requirements. This invention is proposed to address these challenges. Summary of the Invention

[0013] The purpose of this invention is to address how to effectively eliminate or significantly reduce the privacy leakage risk caused by the "distance preservation characteristic" in existing feature transformation methods without sacrificing recognition performance, i.e., to achieve decoupling between the protected template and the original facial features; how to overcome the contradiction between the inherent "ambiguity" of biometric data and the "exact matching" requirement of cryptography, thereby stably generating high-quality, reproducible encryption keys from noisy facial images, while avoiding the vulnerability of relying on traditional error correction codes (ECC); and how to achieve accurate identification of known users and reliable rejection of unknown or impersonating users in open-set scenarios, effectively quantifying the uncertainty of decision-making, and improving the robustness and security of the system.

[0014] To achieve the above objectives, this invention provides a privacy-preserving open-set face recognition method based on LoRa and uncertainty estimation, comprising the following steps:

[0015] S1. Generate a set of hash codes, extract facial features from the pre-trained basic feature encoder, map the facial features to a unique binary hash code in the hash code set, define it as the target encryption key, decouple the potential association between facial features and the target encryption key, and construct the key space.

[0016] S2. Select face images in the key space as query samples, and use low-rank adaptation technology to fine-tune the basic feature encoder multiple times to make it a personalized feature encoder, so as to converge the facial features of the query samples under different conditions to the target encryption key.

[0017] S3. When fine-tuning the basic feature encoder, the total loss is used to learn the evidence value of the output and deduce the degree of "uncertainty" when the personalized feature encoder outputs the prediction key. During the authentication phase, the uncertainty is calculated. If the uncertainty value is greater than the preset uncertainty threshold, the access request is rejected. If the uncertainty value is not greater than the uncertainty threshold, the prediction key is matched bit by bit with the encryption key in the database.

[0018] As a further improvement to this technical solution, the hash code set includes multiple binary hash codes, each of which has the same probability of taking the value of each bit, and the different bits of each binary hash code are independently correlated; and a preset loss threshold is set, wherein the Hamming distance between any two binary hash codes in the hash code set is greater than the preset loss threshold, and the binary hash code allocated to each user is an unoccupied binary hash code.

[0019] The beneficial effect of the above-mentioned further scheme is that it creates a highly secure and irreversible key space, completely decouples the potential association between the protected template and the original facial features, thereby fundamentally eliminating the privacy leakage risk and reconstruction attack threat brought about by the "distance preservation characteristic" in traditional biometric encryption.

[0020] Based on the above technical solution, the present invention can be further improved as follows: pre-training the basic feature encoder:

[0021] The system perceives large-scale face datasets, which contain a large number of face images with different identities, poses, expressions, and lighting conditions; and aligns the face images in the large-scale face datasets, and then standardizes them to a uniform size and pixel range.

[0022] A deep neural network architecture is adopted to extract facial features layer by layer through a multi-layer network structure: the deep neural network architecture includes a bottom layer network, a middle layer network, and a high layer network;

[0023] The bottom layer network consists of multiple convolutional layers, used to learn the basic visual features of face images; the middle layer network integrates the basic visual features into more complex local features through a combination of convolutional and pooling layers; the high layer network further extracts face features directly related to identity; and enhances the expressive power of face features through nonlinear activation functions, and solves the gradient vanishing problem when training the basic feature encoder in deep neural network architecture through residual connection technology.

[0024] As a further improvement to this technical solution, the key space is constructed as follows: after sensing the user registration request, multiple face images corresponding to the user are collected, the basic feature encoder extracts the face features of each face image, the unassigned binary hash code in the hash code set is retrieved and mapped to the face features, and the registration request, the target encryption key and the face features are associated to construct the key space.

[0025] The beneficial effect of the above-mentioned further scheme is that by forcibly mapping continuous, high-dimensional facial features to a discrete, preset binary hash code, a highly irreversible transformation is achieved. This means that even if an attacker obtains the final encryption key, they will not be able to reverse-engineer the original facial features, thus providing industry-leading privacy protection. For system flexibility and scalability, this invention also supports a "one person, one model" strategy, that is, each user has a customized target encryption key.

[0026] Based on the above technical solution, the present invention can be further improved as follows: the working steps of the low-rank adaptation technique for fine-tuning the basic feature encoder are as follows:

[0027] S2.1 Select face images corresponding to the same identity information in the key space as query samples and start training. Insert two low-rank matrices into the key layer of the basic feature encoder to form a new weight matrix and obtain the personalized feature encoder after the first fine-tuning. The personalized feature encoder extracts adaptive features from the query samples.

[0028] S2.2 Input adaptive features to the mapping function, which includes a flip layer, a dropout layer, a fully connected layer and a Softplus activation function. The original evidence value and the prediction key are output through the fully connected layer branch. The original evidence value is then processed by the Softplus activation function to ensure that the output is non-negative, thus obtaining the final evidence value. The prediction key and evidence value are generated simultaneously.

[0029] S2.3 Calculate the total loss after fine-tuning the basic feature encoder: feature space loss + hash space loss + evidence classification loss;

[0030] The feature space loss is the cosine similarity loss between adaptive features and face features; the hash space loss is the matching degree between the predicted key and the target encryption key; the evidence classification loss includes a mean squared error term and a KL divergence regularization term; the mean squared error term is the probability value of the face image corresponding to the adaptive feature belonging to different users, and the KL divergence regularization term is the difference between the probability value of the face image corresponding to the adaptive feature and the probability value of the face image actually belonging to the user;

[0031] S2.4. If the total loss is less than or equal to the preset loss threshold, it is determined that the basic feature encoder still needs to be adjusted, and backpropagation is performed: the output of the mapping function in the personalized feature encoder is backpropagated to the fully connected layer near the output of the basic feature encoder, and the gradients of the prediction key and evidence value with respect to the total loss are calculated: based on the hash space loss and the evidence classification loss, the partial derivatives of the loss with respect to the mapping function are calculated; then the gradients of the two low-rank matrices are calculated: based on the feature space loss and the hash space loss, the partial derivatives of the loss with respect to the low-rank matrix are calculated.

[0032] S2.5. The optimizer updates the weight values ​​of the two initially inserted low-rank matrices according to the gradient descent principle based on the backpropagation gradient: the new weight value is equal to the old weight value minus the learning rate multiplied by the gradient.

[0033] S2.6 Iteration process: Repeat S2.1-S2.5 for each training iteration until the total loss is greater than the preset loss threshold, and obtain the personalized feature encoder after fine-tuning for each query sample.

[0034] The beneficial effect of the above-mentioned further scheme is that, without significantly modifying or retraining the massive basic feature encoder backbone, the behavior of the basic feature encoder can be efficiently adjusted by adding only a small number of trainable low-rank matrices. This enables the basic feature encoder to learn user-specific and robust mapping relationships, accurately converging the user's facial features under different conditions to its preset encryption key.

[0035] Based on the above technical solution, the present invention can be further improved as follows:

[0036] During the authentication phase, an authentication request is received, the input identity information in the authentication request is retrieved, and the key space is compared to see if there is identity information that is the same as the output identity information. If there is identity information in the key space that matches the input identity information, then the personalized feature encoder corresponding to the identity information is retrieved through the real-time captured face image, and adaptive features are extracted through the corresponding personalized feature encoder.

[0037] The adaptive features are then input into the mapping function, which outputs the prediction key and evidence value.

[0038] As a further improvement to this technical solution, when the personalized feature encoder extracts adaptive features, it sums all intermediate evidence values ​​pointing to the same known user to form the total positive class evidence for the known user; it summarizes all intermediate evidence values ​​pointing to users who do not belong to the known user to form the total dummy class evidence, which reflects the degree of support for unknown users; and it uses the dummy class weight divided by the sum of all the total positive class evidence for each category to convert the evidence value into uncertainty.

[0039] The authentication phase involves: setting a preset uncertainty threshold; if the uncertainty is less than or equal to the uncertainty threshold, the authentication request is considered successful; if the uncertainty is greater than the uncertainty threshold, the authentication request is considered unsuccessful; after a successful authentication request, the target encryption key and the predicted key are compared; if the target encryption key equals the predicted key, the authentication is considered successful; if the target encryption key does not equal the predicted key, the authentication is considered unsuccessful.

[0040] As a further improvement to this technical solution, the preset uncertainty threshold is: a perception verification set, which is determined by analyzing the false recognition rate and false rejection rate curves on the verification set.

[0041] As a further improvement to this technical solution, the comparison between the target encryption key and the predicted key is a bit-by-bit exact consistency comparison, rather than a similarity comparison.

[0042] The beneficial effect of the above-mentioned further scheme is that, by jointly optimizing multiple total losses (including the loss to ensure feature quality, the loss to ensure precise key matching, and the evidence loss to quantify uncertainty in the model), the deep learning feature encoder can not only accurately generate the user's encryption key, but also clearly express its "uncertain" state when faced with unregistered users or abnormal input.

[0043] Meanwhile, during the authentication phase, this invention employs a unique decision-making process of "uncertainty filtering first, then precise key matching": First, a judgment is made based on the uncertainty. If the value exceeds a preset threshold, it indicates uncertainty about the input (e.g., it is an unknown user, an anomalous sample, or an attack), thus directly rejecting the access request. Only when the uncertainty is less than or equal to the threshold (indicating sufficient confidence in its identity prediction) will the second step be initiated, namely, performing a bit-by-bit precise matching between the target encryption key and the predicted key. Only when the two are completely identical will the authentication be passed. This dual decision-making mechanism greatly enhances the robustness, security, and practicality of the system in open set environments, effectively preventing unknown users from bypassing the system.

[0044] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the figures. Attached Figure Description

[0045] Figure 1 This is a flowchart of the working steps of the present invention;

[0046] Figure 2 This is a schematic diagram of the key space construction of the present invention;

[0047] Figure 3 A flowchart illustrating the steps involved in constructing the personalized feature encoder for this invention;

[0048] Figure 4 This is a flowchart comparing the certification stages of this invention. Detailed Implementation

[0049] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] refer to Figures 1-4 As shown, the privacy-preserving open-set face recognition method based on LoRa and uncertainty estimation includes the following steps:

[0051] like Figure 1 S1, Generate a set of hash codes Assign a binary hash code to each user The target encryption key for identity information And construct the key space:

[0052] Generate hash code set The specific work steps are as follows:

[0053] S1.1, Hash Code Set It includes multiple binary hash codes. ( , (the number of multiple binary hash codes), each binary hash code Each of them (Binary hash code) The Bit( , The probability of each value (0 or 1) being the length of the binary hash code is the same, and each binary hash code... Different bits are independent of each other, thus ensuring the binary hash code. Partially distributed in binary space—no binary hash codes will appear. This avoids situations where a certain bit in the binary hash code remains 0 or 1 for an extended period, thus preventing binary hash codes from being manipulated in the first place. The risk of collisions arises from bias in the hash code (two different users have highly similar hash codes).

[0054] S1.2, Preset distance threshold (The distance threshold is a quality control parameter used to ensure that the minimum Hamming distance between all code pairs in the hash code set meets the minimum difference requirement of the application. The specific value depends on the application requirements. For example, a smaller distance threshold is suitable for scenarios with short hash code lengths or low requirements for distinguishability, while a larger distance threshold is suitable for scenarios requiring high distinguishability and strong fault tolerance.) Any two binary hash codes and binary hash code ( Hamming distance between ≥ Distance threshold This ensures the set of hash codes Multiple binary hash codes There are clear distinguishable features between them;

[0055] Hash code set The generation of hash codes can be achieved through any method that can generate a set of binary vectors that are uniformly distributed in binary space and sufficiently far apart from each other, such as generating hash code sets based on random projection, PCA dimensionality reduction followed by quantization, or more complex optimization algorithms. ;

[0056] S1.3 After receiving a user registration request, and the registration request includes identity information (such as user ID, name, etc.), retrieve the hash code from the set. In, for each user's hash code set A unique, unused binary hash code The target encryption key for identity information And the assigned binary hash code It is permanent and unique.

[0057] The working principle of constructing the key space in S1 is as follows: a large-scale face dataset is perceived (using publicly available datasets such as LFW (Labeled Faces in the Wild), AgeDB-30, and CFP (CelebFaces Attributes) for training and evaluation) to pre-train the basic feature encoder. (e.g., ArcFace, MagFace, etc.) It perceives multiple (e.g., 5-10) facial images (each image representing a face from a different angle, with different expressions, and under different lighting conditions) captured by a camera or other biometric acquisition device (e.g., a high-resolution camera), and employs a basic feature encoder. Extract the original deep facial features from each face image. ;

[0058] Step 2: Create a registration request and the corresponding target encryption key. and facial features The mapping relationship between them, and the registration request and target encryption key that have the mapping relationship. and facial features Constructed as a key space;

[0059] By establishing a mapping relationship, a highly irreversible transformation is achieved, meaning that even if an attacker obtains the final target encryption key... It is also impossible to infer facial features from them. This provides industry-leading privacy protection.

[0060] Pre-trained basic feature encoder The working principle is as follows: It perceives a large-scale face dataset, which contains a large number of face images with different identities, poses, expressions and lighting conditions; and aligns the face images in the large-scale face dataset, and then standardizes them to a uniform size and pixel range.

[0061] A deep neural network architecture is adopted to extract facial features layer by layer through a multi-layer network structure: the deep neural network architecture includes a bottom layer network, a middle layer network, and a high layer network;

[0062] The bottom-layer network consists of multiple convolutional layers, used to learn the basic visual features of face images; the middle-layer network integrates the basic visual features into more complex local features through a combination of convolutional and pooling layers; the high-layer network further extracts facial features directly related to identity; and enhances the expressive power of facial features through non-linear activation functions, and solves the problem of training the basic feature encoder in the deep neural network architecture through residual connection techniques. The problem of vanishing temporal gradients.

[0063] like Figure 2 When constructing the key space, the basic feature encoder in step one Specifically, it adopts an industry-leading pre-trained basic feature encoder. As the backbone, such as ArcFace based on ResNet50, and others such as MagFace, AdaFace, PartialFace, DCTDP, etc.; the feature dimension of these models is usually 512-dimensional.

[0064] Furthermore, publicly available datasets such as LFW (Labeled Faces in the Wild), AgeDB-30, and CFP (CelebFacesAttributes) were used to train and evaluate the basic feature encoder. To simulate an open set scenario, face images from the CelebA dataset were additionally introduced as "unknown" interference samples; ensuring that the ratio of known to unknown samples in each face dataset matches the actual open set scenario (e.g., 95.24% open set ratio).

[0065] And the pre-trained basic feature encoder Besides models like ArcFace and MagFace, other advanced face recognition backbone networks, such as MobileFaceNet and EfficientNet, can be used, or feature extractors pre-trained using self-supervised learning and contrastive learning methods can be combined; only the basic feature encoder is required. Capable of extracting facial features with sufficient discriminative power That's all.

[0066] To enable the basic feature encoder Able to adapt to facial features Mapped to the target encryption key assigned to the user S2. Use Low-Rank Adaptation (LoRA) parameters for the feature encoder. Fine-tuning yields a personalized feature encoder ,like Figure 3 The specific working principle is as follows:

[0067] S2.1 Select face images (multiple face images with different conditions) corresponding to the same identity information in the key space as query samples. To start training, first calculate the total loss. (Total loss) For feature space loss + Hash space loss +Evidence Classification Loss Two low-rank matrices are inserted into the fully connected layer near the output of the basic feature encoder (referring to the layer in the network that is most sensitive to and discriminative of feature representation, such as the fully connected layer near the output, which directly determines the dimension of the final feature). and This forms a new weight matrix, resulting in the personalized feature encoder after initial fine-tuning. Personalized feature encoder Extract adaptive features from the query sample:

[0068] low-rank matrix and Low- and medium-rank matrices The dimension is Used to map high-dimensional features to a low-dimensional space; low-rank matrix The dimension is This is used to map low-dimensional features back to the original dimension and is superimposed on the output of the fully connected layer near the output of the basic feature encoder. Facial feature dimension (basic feature encoder) Output facial features (dimensions) A low-rank matrix and The low-rank parameter, r, is crucial and is typically much smaller than that of the basic feature encoder. The dimension of the original matrix: For example, for a 512-dimensional feature, the low-rank parameter r can be set to a small value such as 4, 8, or 16, ensuring lightweight fine-tuning;

[0069] S2.2 Input adaptive features to the mapping function, which includes a flip layer, a dropout layer, a fully connected layer and a Softplus activation function. The original evidence value and the prediction key are output through the fully connected layer branch. The original evidence value is then processed by the Softplus activation function to ensure that the output is non-negative, thus obtaining the final evidence value. The prediction key and evidence value are generated simultaneously.

[0070] S2.3 Calculate and fine-tune the basic feature encoder The final total loss is: feature space loss + hash space loss + evidence classification loss;

[0071] The feature space loss is the cosine similarity loss between adaptive features and face features; the hash space loss is the matching degree between the predicted key and the target encryption key; the evidence classification loss includes a mean squared error term and a KL divergence regularization term; the mean squared error term is the probability value of the face image corresponding to the adaptive feature belonging to different users, and the KL divergence regularization term is the difference between the probability value of the face image corresponding to the adaptive feature and the probability value of the face image actually belonging to the user;

[0072] S2.4. Preset loss threshold: If the total loss is less than or equal to the preset loss threshold, then determine the basic feature encoder. Further adjustments and backpropagation are needed: personalization feature encoder. The output of the mapping function is backpropagated to the basic feature encoder. Near the output of the fully connected layer, calculate the gradients of the predicted key and evidence value with respect to the total loss: based on the hash space loss and the evidence classification loss, calculate the partial derivatives of the loss with respect to the mapping function; then calculate the gradients of the two inserted low-rank matrices: based on the feature space loss and the hash space loss, calculate the partial derivatives of the loss with respect to the low-rank matrix.

[0073] S2.5. The optimizer updates the weight values ​​of the two initially inserted low-rank matrices according to the gradient descent principle based on the backpropagation gradient: the new weight value is equal to the old weight value minus the learning rate multiplied by the gradient.

[0074] S2.6 Iterative Process: Repeat S2.1-S2.5 for each training iteration until the total loss exceeds the preset loss threshold, thus obtaining the personalized feature encoder after fine-tuning for each query sample. .

[0075] Personalized feature encoder Fine-tuning alternatives: In addition to the standard Low-Rank Adaptation (LoRA) parameter structure, other efficient fine-tuning techniques can be explored, such as Prefix-Tuning, Prompt-Tuning, or Adapter-based methods. These methods also aim to efficiently adapt large pre-trained personalized feature encoders with a small number of parameters. .

[0076] like Figure 4 S3. During the authentication phase, the system receives the authentication request, retrieves the input identity information from the authentication request, and compares it with the key space to see if the same identity information exists as the output identity information. If the key space contains identity information that matches the output identity information, the system then uses a camera or other biometric acquisition device (e.g., a high-resolution camera) to capture a face image in real time and passes it through a corresponding personalized feature encoder. Extracting adaptive features ;

[0077] Adaptive characteristics Input to mapping function (mapping function) The network structure includes flipped layers, dropout layers, fully connected layers, and the Softplus activation function. The network structure can be adjusted according to actual needs, such as increasing or decreasing the number of layers, using different activation functions, etc., to optimize performance. (Mapping function) Output prediction key And the evidence value e, and the corresponding steps:

[0078] Step 1: Receive adaptive characteristics as (in For adaptive characteristics (feature dimension, such as 512 dimensions), mapping function The operations for each layer are as follows: The flipped layer has the following adaptive characteristics. Feature Dimensions Perform a random rearrangement and output the flipped features: in This is a random dimension flipping operation (such as swapping dimensions by randomly arranging a matrix).

[0079] Dropout layer uses probability (e.g., 0.5) Randomly discard adaptive features Output of some dimensions in the data: in For Dropout operations, the discarded dimension values ​​are set to 0;

[0080] Step 2: Combining Fully Connected Layers (Feature Compression and Branching):

[0081] The first fully connected layer (ReLU activation): converts features From the feature dimension Compress to Dimensions (e.g., 256 dimensions): ,in This is the weight matrix. For bias;

[0082] The second fully connected layer (divided into two branches):

[0083] Key generation branch: Compressed to hash code length Dimensionality (e.g., 128-dimensional), output continuous value vector : , ( )in , For key branch parameters;

[0084] Evidence value generation branch: Outputs 1D original evidence value ,in , For evidence branch parameters;

[0085] Step 3: Predict the key The final expression for the evidence value e:

[0086] Prediction key (discretization): For continuous vectors Hard quantization is performed to obtain 0 / 1 discrete hash codes. : ,in for The k-th dimension, with a threshold of 0.5, ensures the uniqueness of the discretization result;

[0087] Evidence value (Softplus activation): For the original evidence value Activate with Softplus to ensure non-negative output: .

[0088] Receive mapping function Output prediction key And the evidence value e; using deep learning of evidence (based on Dirichlet distribution) to transform the evidence value e into uncertainty. ,in These are dummy weights, used to enhance the modeling ability for unknown categories. This is the accumulated evidence (including prior knowledge) that "the input belongs to class j".

[0089] Preset uncertainty threshold , will uncertainty With uncertainty threshold In comparison, the preset uncertainty threshold The working principle is as follows: A validation set (whose source is similar to face datasets, such as the publicly available datasets mentioned above (LFW, AgeDB-30, CFP, etc.), or specific face data collected based on actual application scenarios) is used to determine the uncertainty threshold by analyzing the false recognition rate (the probability of an unregistered user or malicious attacker being mistaken for a legitimate user) and false rejection rate (the probability of a legitimate registered user being mistakenly identified as an unknown user and rejected) curves on the validation set. It iterates through the false acceptance and false rejection rates corresponding to the candidate thresholds, finding the equilibrium point on the false acceptance and false rejection rate curves, or selecting an uncertainty threshold based on requirements to achieve a balance between security and availability, such as an uncertainty threshold. =0.15;

[0090] Compared to traditional open set identification based on similarity thresholds, this invention provides a more refined and reliable unknown sample discrimination mechanism, which significantly enhances the security and practicality of the system in real open environments.

[0091] And retrieve the corresponding target encryption key from the key space according to the authentication request. and the target encryption key With prediction key In comparison, user identity is determined through two different stages:

[0092] Phase 1: Uncertainty Filtering: If uncertainty... Greater than the uncertainty threshold This indicates that the personalized feature encoder The identity of the current facial image (the facial image captured when the user inputs information (before authentication)) is extremely uncertain, and this situation usually occurs in the following circumstances:

[0093] The face does not belong to any registered user (unknown user), the face is fake (liveness attack), and the face image quality is extremely poor, causing issues with the personalized feature encoder. Unable to be effectively identified;

[0094] At the same time, it is judged as an authentication failure and access is denied, thus effectively preventing unknown or malicious users from entering the subsequent precise matching process;

[0095] If uncertainty Less than or equal to the uncertainty threshold This indicates that the personalized feature encoder If there is sufficient confidence in the identity of the current face image, it can be determined that the authentication request is likely a legitimate authentication request from a registered user, and then proceed to the next stage of precise key matching;

[0096] Phase Two: Precise Key Matching: Prerequisite: Performed only after the uncertainty filtering in Phase One has passed; Judgment Logic:

[0097] Adaptive characteristics (After the authentication request is entered, the captured facial image is processed by a personalized feature encoder) Extracted adaptive features The corresponding target encryption key With personalized feature encoder Output prediction key Compare each element individually;

[0098] If the target encryption key Equal to prediction key This indicates the target encryption key. With prediction key A perfect match indicates successful authentication.

[0099] If the target encryption key Not equal to the prediction key (Even if only one character is different): This indicates the target encryption key With prediction key Incomplete match, authentication failed.

[0100] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A privacy-preserving open-set face recognition method based on LoRa and uncertainty estimation, characterized in that, Includes the following steps: S1. Generate a set of hash codes, extract facial features from the pre-trained basic feature encoder, map the facial features to a unique binary hash code in the hash code set, define it as the target encryption key, decouple the potential association between facial features and the target encryption key, and construct the key space. S2. Select face images in the key space as query samples, and use low-rank adaptation technology to fine-tune the basic feature encoder multiple times to make it a personalized feature encoder, so as to converge the facial features of the query samples under different conditions to the target encryption key. The working steps of the low-rank adaptive technique for repeatedly fine-tuning the basic feature encoder into a personalized feature encoder are as follows: S2.1 Select face images corresponding to the same identity information in the key space as query samples and start training. Insert two low-rank matrices into the fully connected layer near the output of the basic feature encoder to form a new weight matrix and obtain the personalized feature encoder after initial fine-tuning. The personalized feature encoder extracts adaptive features from the query samples. S2.2 Input adaptive features to the mapping function, which includes a flip layer, a dropout layer, a fully connected layer and a Softplus activation function. The original evidence value and the prediction key are output through the fully connected layer branch. The original evidence value is then processed by the Softplus activation function to ensure that the output is non-negative, thus obtaining the final evidence value. The prediction key and evidence value are generated simultaneously. S2.3 Calculate the total loss after fine-tuning the basic feature encoder: the sum of feature space loss, hash space loss, and evidence classification loss; The feature space loss is the cosine similarity loss between adaptive features and face features; the hash space loss is the matching degree between the predicted key and the target encryption key; the evidence classification loss includes a mean squared error term and a KL divergence regularization term; the mean squared error term is the probability value of the face image corresponding to the adaptive feature belonging to different users, and the KL divergence regularization term is the difference between the probability value of the face image corresponding to the adaptive feature and the probability value of the face image actually belonging to the user; S2.

4. If the total loss is less than or equal to the preset loss threshold, it is determined that the basic feature encoder still needs to be adjusted and backpropagation is performed: the output of the mapping function in the personalized feature encoder is backpropagated to the fully connected layer near the output of the basic feature encoder, and the gradients of the prediction key and evidence value with respect to the total loss are calculated: based on the hash space loss and the evidence classification loss, the partial derivatives of the loss with respect to the mapping function are calculated. Next, calculate the gradient of inserting two low-rank matrices: based on the feature space loss and hash space loss, calculate the partial derivative of the loss with respect to the low-rank matrix; S2.

5. The optimizer updates the weight values ​​of the two initially inserted low-rank matrices according to the gradient descent principle based on the backpropagation gradient: the new weight value is equal to the old weight value minus the learning rate multiplied by the gradient. S2.6 Iteration process: Repeat S2.1-S2.5 for each training iteration until the total loss is greater than the preset loss threshold, and obtain the personalized feature encoder after fine-tuning for each query sample; S3. When fine-tuning the basic feature encoder, the total loss is used to learn the evidence value of the output and deduce the degree of "uncertainty" when the personalized feature encoder outputs the prediction key. During the authentication phase, the uncertainty is calculated. If the uncertainty value is greater than the preset uncertainty threshold, the access request is rejected. If the uncertainty value is not greater than the uncertainty threshold, the prediction key is matched bit by bit with the encryption key in the database.

2. The privacy-preserving open-set face recognition method based on LoRa and uncertainty estimation according to claim 1, characterized in that: The hash code set includes multiple binary hash codes, each of which has the same probability of taking the value of each bit, and the different bits of each binary hash code are independent of each other; and a preset distance threshold is set, such that the Hamming distance between any two binary hash codes in the hash code set is greater than the distance threshold, and the binary hash code assigned to each user is an unoccupied binary hash code.

3. The privacy-preserving open-set face recognition method based on LoRa and uncertainty estimation according to claim 2, characterized in that: The pre-trained basic feature encoder perceives a large-scale face dataset, which contains a large number of face images with different identities, poses, expressions, and lighting conditions; and aligns the face images in the large-scale face dataset, and then normalizes them to a uniform size and pixel range. A deep neural network architecture is adopted to extract facial features layer by layer through a multi-layer network structure: the deep neural network architecture includes a bottom layer network, a middle layer network, and a high layer network; The underlying network consists of multiple convolutional layers, used to learn the basic visual features of face images; The middle-layer network integrates basic visual features into more complex local features through a combination of convolutional and pooling layers; The high-level network further extracts facial features directly related to identity; and enhances the expressive power of facial features through nonlinear activation functions, and solves the gradient vanishing problem when training the basic feature encoder in a deep neural network architecture through residual connection technology.

4. The privacy-preserving open set face recognition method based on LoRa and uncertainty estimation according to claim 2, characterized in that: Constructing the key space: After sensing the user registration request, multiple face images corresponding to the user are collected. The basic feature encoder extracts the face features of each face image, retrieves the unassigned binary hash code from the hash code set, maps it to the face features, and associates the registration request, the target encryption key, and the face features to construct the key space.

5. The privacy-preserving open-set face recognition method based on LoRa and uncertainty estimation according to claim 1, characterized in that: During the authentication phase, an authentication request is received, the input identity information in the authentication request is retrieved, and the key space is compared to see if there is identity information that is the same as the output identity information. If there is identity information in the key space that matches the input identity information, then the personalized feature encoder corresponding to the identity information is retrieved through the real-time captured face image, and adaptive features are extracted through the corresponding personalized feature encoder. The adaptive features are then input into the mapping function, which outputs the prediction key and evidence value.

6. The privacy-preserving open-set face recognition method based on LoRa and uncertainty estimation according to claim 5, characterized in that: When the personalized feature encoder extracts adaptive features, it sums all intermediate evidence values ​​pointing to the same known user to form the total positive class evidence for the known user; it summarizes all intermediate evidence values ​​pointing to users who do not belong to the known user to form the total dummy class evidence, which reflects the degree of support for unknown users; and it uses the dummy class weight divided by the sum of the total positive class evidence for all inputs belonging to each category to convert the evidence value into uncertainty.

7. The privacy-preserving open-set face recognition method based on LoRa and uncertainty estimation according to claim 1, characterized in that: The authentication phase includes: a preset uncertainty threshold; If the uncertainty is less than or equal to the uncertainty threshold, the authentication request is considered successful; if the uncertainty is greater than the uncertainty threshold, the authentication request is considered unsuccessful. After the authentication request is successful, the target encryption key and the predicted key are compared. If the target encryption key is equal to the predicted key, the authentication is considered successful; if the target encryption key is not equal to the predicted key, the authentication is considered to have failed.

8. The privacy-preserving open-set face recognition method based on LoRa and uncertainty estimation according to claim 7, characterized in that: The preset uncertainty threshold includes the following steps: The perception verification set is used to analyze the false recognition rate and false rejection rate curves. The false recognition rate is the probability that an unregistered user or a malicious attacker is mistakenly identified as a legitimate user, and the false rejection rate is the probability that a legitimate registered user is mistakenly identified as an unknown user and rejected. The false recognition rate and false rejection rate corresponding to the candidate thresholds are traversed, and the equilibrium point on the curves of the false recognition rate and false rejection rate is found, or the candidate threshold is selected as the uncertainty threshold according to the requirements.

9. The privacy-preserving open set face recognition method based on Lora and uncertainty estimation according to claim 8, wherein the comparison between the target encryption key and the predicted key is a bitwise complete consistency comparison, rather than a similarity comparison.

Images