Privacy protection open set face recognition method based on Lora and uncertainty estimation

Abstract

The invention relates to the technical field of information security and artificial intelligence, in particular to a privacy protection open-set face recognition method based on Lora and uncertainty estimation, which comprises the following steps: generating a hash code set, pre-training a basic feature encoder to extract face features, mapping the face features to binary hash codes in the hash code set, and performing face recognition on the binary hash codes. Defining the key as a target encryption key, and constructing a key space; selecting a query sample, finely adjusting the basic feature encoder for multiple times as a personalized feature encoder, and converging facial features of the query sample under different conditions to a mapped target encryption key; and when the basic feature encoder is finely adjusted, combining the total loss to learn an evidence value about output, in an authentication stage, calculating uncertainty, if the uncertainty value is greater than an uncertainty threshold value, refusing an access request, and if the uncertainty value is not greater than a pre-uncertainty threshold value, executing bit-by-bit matching of a prediction key and an encryption key in a database.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information security and artificial intelligence, in particular to a privacy protection open set face recognition method based on Lora and uncertainty estimation. BACKGROUND

[0002] With the wide application of face recognition (FR) technology in finance, security, smart home, medical health and many other fields, the sensitive personal information contained in the face image and feature data on which it depends has caused growing privacy concerns. Face data not only can directly identify individual identity, but also can reveal sensitive attributes that can be visually identified, including but not limited to gender, age, race, health status, etc. Therefore, how to effectively protect user privacy while fully utilizing the convenience of face recognition has become a key challenge faced by the development and deployment of FR systems. Existing privacy protection face recognition methods can be mainly summarized into three categories: visual removal methods, feature transformation methods and biometric encryption methods.

[0003] (1) Visual removal methods: Principle: This kind of method aims to eliminate or mask the visual sensitive information in the face image through image processing techniques such as anonymization, blurring, face key area removal or replacement. For example, by decoupling the visual-related and identity-related components, and training the model on the "purified" face data.

[0004] Advantages: It can intuitively reduce the risk of image privacy leakage, making it difficult for unauthorized personnel to visually identify individuals.

[0005] Disadvantages: ① Insufficient template security: Although part of the visual information is removed, the face template generated by these methods usually still retains rich identity information. These templates, if obtained by malicious attackers during storage or transmission, may still face the risk of reconstruction attack. Attackers can use advanced generation models to reverse engineer the original face image from the protected template, resulting in privacy leakage. ② High training cost: Most visual removal methods require retraining of the FR model to adapt to the processing of images with less sensitive information. This not only has high computational cost, but also may be difficult to apply flexibly in actual deployment. ③ Incomplete protection: Simply eliminating visual clues cannot completely solve the problem of sensitive attribute inference, nor can it provide encryption-level protection for the final generated feature template.

[0006] (2) Feature Transformation Methods: Principle: These methods focus on performing irreversible transformations (such as hashing, encryption, or random projections) on the feature vectors extracted from face images, thereby generating secure biometric templates. Examples include BioHash, MLPHash, WiFaKey, etc.

[0007] Advantages: Aim to provide stronger security guarantees, once the transformation is completed, the original features are difficult to recover from the transformed templates, thereby enhancing privacy protection.

[0008] Disadvantages: ① Performance degradation: In order to achieve the irreversibility of transformation, these methods often introduce noise or information loss in the transformation process, which may lead to a decrease in the accuracy of face recognition. There is an inherent trade-off between privacy protection and recognition performance. ② Risk of distance-preserving property: Many feature transformation methods strive to maintain the relative distance consistency between the original feature space and the transformed feature space in order to maintain recognition performance. That is, if the original feature A is closer to C than B, then the transformed A' will also be closer to C' than B'. This "distance-preserving" property, while beneficial for recognition, becomes a potential weakness for reconstruction attacks. Attackers can use this property to iteratively optimize and gradually approach the original face, thereby bypassing privacy protection mechanisms.

[0009] (3) Biometric Encryption Methods: Principle: These methods aim to combine the fuzziness of biometric features with the precision of cryptographic keys, generating or securely releasing an encrypted key from biometric data through various techniques (such as Fuzzy Extractor, Fuzzy Vault, Key Binding, etc.). These methods usually do not directly transform the original features for matching, but use biometric features as a credential to generate or unlock an encrypted key, thereby achieving authentication.

[0010] Advantages: ① Can combine the authentication ability of biometric features with the strict security of cryptography, making the identity verification process closely integrated with encryption principles (such as data encryption, digital signature). ② If the key generation is successful, the generated key can be directly used for cryptographic applications such as encryption, signature, etc., providing a higher level of security. ③ Through the revocation and update of the key, the revocation of the biometric credential can be theoretically realized.

[0011] Disadvantages: ① Closed-set scenario limitation: Existing hash-based biometric methods are usually designed for "closed-set" scenarios, i.e., the system only recognizes registered users, and cannot effectively handle or reject unregistered "unknown" identities. In actual open environments, FR systems need to have the ability to recognize known users and reject unknown users. ② Dependence on Error Correction Codes (ECC) and its vulnerability: Biometric data has inherent "fuzziness" or variability, i.e., there are subtle differences between different samples of the same person. Cryptography requires precise "one-to-one" matching. In order to bridge this gap, many existing biometric encryption systems rely on ECC to correct small errors in the matching process. However, this approach introduces new vulnerabilities, such as attackers may attack by analyzing or manipulating the error correction codebook. In addition, the fixed error boundary also limits its flexibility in highly variable open-set environments. ③ Key generation challenge: How to generate stable, unique encryption keys from high-dimensional, continuous and inherently variable biometric data, while meeting the requirements of irreversibility, determinism and collision resistance of cryptography, is a major technical challenge.

[0012] In summary, existing privacy-preserving face recognition methods still have significant limitations in privacy protection strength, recognition performance, open-set adaptability, and attack resistance. Visual removal methods have shortcomings in template security; feature conversion methods face performance degradation, reconstruction risk caused by distance preservation characteristics, closed-set limitations, and dependence on vulnerable ECC; and biometric encryption methods struggle between key stability, auxiliary data privacy risk, and performance and security trade-offs. Therefore, there is an urgent need for a new method that can achieve a better balance between strong privacy protection, high recognition accuracy, and open-set adaptability, and effectively resist known and potential attacks to meet the growing market demand and strict regulatory requirements. The present invention is proposed to solve the above challenges. SUMMARY

[0013] The purpose of the present invention is to effectively eliminate or significantly weaken the privacy leakage risk brought by the "distance preservation characteristic" in existing feature conversion methods without sacrificing recognition performance, i.e., to achieve decoupling of the protected template and the original facial feature; to overcome the contradiction between the inherent "fuzziness" of biometric data and the requirement of "precise matching" in cryptography, thereby stably generating high-quality, reproducible encryption keys from noisy facial images, while avoiding the vulnerability of traditional Error Correction Codes (ECC); to achieve accurate recognition of known users and reliable rejection of unknown or impersonated users in open-set scenarios, and effectively quantify the uncertainty of decision-making, improving the robustness and security of the system.

[0014] To achieve the above object, the application provides a privacy protection open set face recognition method based on Lora and uncertainty estimation, comprising the following steps: S1, generating a hash code set, pre-training a basic feature encoder to extract face features, mapping the face features to a unique binary hash code in the hash code set, defining it as a target encryption key, decoupling the potential association between the face features and the target encryption key, and constructing a key space; S2, selecting a face image in the key space as a query sample, using low-rank adaptive technology to fine-tune the basic feature encoder to an individualized feature encoder, and converging the face features of the query sample under different conditions to the mapped target encryption key; S3, when fine-tuning the basic feature encoder, learning the evidence value about the output jointly with the total loss, and deriving the "uncertainty" degree when the individualized feature encoder outputs the predicted key; in the authentication stage, the uncertainty is calculated, if the uncertainty value is greater than the preset uncertainty threshold, the access request is rejected, if the uncertainty value is not greater than the uncertainty threshold, the predicted key is matched with the encryption key in the database bit by bit.

[0015] As a further improvement of the technical solution, the hash code set includes a plurality of binary hash codes, the value probability of each bit in each binary hash code is the same, and the different bits in each binary hash code are related and independent; and a loss threshold is preset, the Hamming distance between any two binary hash codes in the hash code set is greater than the preset loss threshold, and the binary hash code allocated for each user is an unoccupied binary hash code.

[0016] The beneficial effect of the above further solution is to create a highly secure and irreversible key space, completely decoupling the potential association between the protected template and the original face features, thereby fundamentally eliminating the privacy leakage risk and reconstruction attack threat caused by the "distance preserving property" in traditional biometric encryption.

[0017] On the basis of the above technical solution, the application can also be improved as follows: the basic feature encoder is pre-trained: perceiving a large-scale face data set, the large-scale face data set containing a large number of face images under different identities, different poses, expressions and illumination conditions; and aligning the face images in the large-scale face data set, and then standardizing them to a uniform size and pixel range; using a deep neural network architecture to extract face features layer by layer through a multi-layer network structure: the deep neural network architecture includes a bottom layer network, a middle layer network and a high layer network; The bottom layer network is composed of multiple convolutional layers, and is used for learning basic visual features of the face image; the middle layer network integrates the basic visual features into more complex local features through the combination of the convolutional layer and the pooling layer; the high layer network further extracts the face features directly related to the identity; and the expression ability of the face features is enhanced through the nonlinear activation function, and the gradient disappearance problem in training the basic feature encoder in the deep neural network architecture is solved through the residual connection technology.

[0018] As a further improvement of the technical solution, the key space is constructed: after perceiving the user registration request, multiple face images corresponding to the user are collected, the face features of each face image are extracted by the basic feature encoder, the unallocated binary hash code in the hash code set is called out and mapped to the face features, and the registration request, the target encryption key and the face features are associated to construct the key space.

[0019] The beneficial effect of the above-mentioned further solution is that by forcibly mapping the continuous and high-dimensional face features to a discrete and preset binary hash code, a highly irreversible conversion is realized, which means that even if an attacker obtains the final encryption key, the original face features cannot be inferred from the final encryption key, thereby providing industry-leading privacy protection; in order to improve the flexibility and scalability of the system, the present application also supports the strategy of "one person one model", that is, each user has a customized target encryption key.

[0020] On the basis of the above technical solution, the present application can also be improved as follows: the low-rank adaptive technology fine-tunes the working steps of the basic feature encoder as follows: S2.1, selecting the face images corresponding to the same identity information in the key space as query samples, and starting training, inserting two low-rank matrices in the key layer of the basic feature encoder to form a new weight matrix, obtaining the personalized feature encoder after the first fine-tuning, and the personalized feature encoder extracts the adaptive features in the query samples; S2.2, input the adaptive features to the mapping function, the mapping function includes a flip layer, a dropout layer, a full connection layer and a Softplus activation function, the original evidence value and the predicted key are output through the full connection layer branch, the original evidence value is processed through the Softplus activation function to ensure that the output is a non-negative value, the final evidence value is obtained, and the predicted key and the evidence value are generated synchronously; S2.3, calculating the total loss after fine-tuning the basic feature encoder: feature space loss+hash space loss+evidence classification loss; The characteristic space loss is a cosine similarity loss between the adaptive feature and the face feature; the hash space loss is a matching degree between the predicted key and the target encryption key; the evidence classification loss includes a mean square error term and a KL divergence regularization term; the mean square error term is a probability value of the face image corresponding to the adaptive feature belonging to different users, and the KL divergence regularization term is a difference between the probability value of the face image corresponding to the adaptive feature and a probability value of the face image actually belonging to a user; S2.4, a preset loss threshold is set, if the total loss is less than or equal to the preset loss threshold, it is judged that the basic feature encoder still needs to be adjusted, and back propagation is performed: the output of the mapping function in the personalized feature encoder is reversely transmitted to the fully connected layer close to the output of the basic feature encoder, and the gradient of the total loss on the predicted key and the evidence value is calculated: the partial derivative of the loss on the mapping function is calculated based on the hash space loss and the evidence classification loss; then the gradient of the two inserted low-rank matrices is calculated: the partial derivative of the loss on the low-rank matrix is calculated based on the characteristic space loss and the hash space loss; S2.5, an optimizer is used to update the weight value of the two inserted low-rank matrices according to the back propagation gradient according to the gradient descent principle: the weight new value is equal to the weight old value minus the learning rate multiplied by the gradient; S2.6, iteration process: S2.1-S2.5 are repeated every time the training iteration, until the total loss is greater than the preset loss threshold, and the personalized feature encoder of each query sample is obtained after fine tuning.

[0021] The beneficial effects of the above further scheme are that the behavior of the basic feature encoder can be efficiently adjusted by adding a small amount of trainable low-rank matrix without greatly modifying or retraining the huge basic feature encoder backbone, so that the basic feature encoder can learn user-specific and robust mapping relationships to accurately converge the face features of the user under different conditions to the preset encryption key.

[0022] On the basis of the above technical scheme, the application can also be improved as follows: In the authentication stage, the input identity information in the authentication request is called out, and it is compared whether there is identity information same as the output identity information in the key space, if there is identity information matching the input identity information in the key space, then the real-time collected face image is used, and then the personalized feature encoder corresponding to the identity information is called out, and the adaptive feature is extracted through the corresponding personalized feature encoder; And the adaptive feature is input into the mapping function, and the mapping function outputs the predicted key and the evidence value.

[0023] As a further improvement of the technical solution, when the personalized feature encoder extracts adaptive features, all intermediate evidence values pointing to the same known user are summed to form total positive class evidence of the known user; all intermediate evidence values pointing to users not belonging to the known user are aggregated to form total negative class evidence, reflecting the support degree of unknown users; and the total negative class weight is divided by the sum of all input total positive class evidence belonging to each class, so as to convert the evidence value into uncertainty.

[0024] The authentication stage: a preset uncertainty threshold; if the uncertainty is less than or equal to the uncertainty threshold, it is judged that the authentication request is successful, and if the uncertainty is greater than the uncertainty threshold, it is judged that the authentication request fails; after the authentication request is successful, the target encryption key is compared with the predicted key; if the target encryption key is equal to the predicted key, it is judged that the authentication is successful; and if the target encryption key is not equal to the predicted key, it is judged that the authentication fails.

[0025] As a further improvement of the technical solution, the preset uncertainty threshold: perception verification set, by analyzing the false rejection rate and false acceptance rate curve on the verification set, the uncertainty threshold is determined.

[0026] As a further improvement of the technical solution, the comparison between the target encryption key and the predicted key is a bit-by-bit complete consistency comparison, rather than a similarity comparison.

[0027] The beneficial effects of the above further solution are that by jointly optimizing multiple total losses (including losses for ensuring feature quality, losses for ensuring key accurate matching, and evidence losses for promoting model quantitative uncertainty), the deep learning feature encoder can not only accurately generate the encryption key of the user, but also clearly express the "uncertain" state when facing non-registered users or abnormal inputs; Meanwhile, in the authentication stage, the present application adopts a unique decision-making process of "first uncertainty filtering, then accurate key matching": first, according to the uncertainty, if the value exceeds the preset threshold, it indicates that the input is uncertain (for example, it is an unknown user, an abnormal sample or an attack), so the access request is directly rejected; only when the uncertainty is less than or equal to the threshold (indicating that the identity prediction is confident enough), the second step is entered, that is, the bit-by-bit accurate matching of the target encryption key and the predicted key is performed, and only when the two are completely consistent, the authentication is passed. Through this double decision-making mechanism, the robustness, security and practicality of the system in the open set environment are greatly improved, and unknown users are effectively prevented from bypassing the system.

[0028] In addition to the purposes, features and advantages described above, the present application has other purposes, features and advantages. The present application will be further described in detail below with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 Flow chart of working steps of the present application; Figure 2 Schematic diagram of key space construction of the present application; Figure 3 Flow chart of working steps of the personalized feature encoder of the present application; Figure 4 Comparison flow chart of the authentication stage of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0031] Reference Figures 1-4 As shown in the figure, the privacy protection open set face recognition method based on Lora and uncertainty estimation includes the following steps: As shown in the figure, the privacy protection open set face recognition method based on Lora and uncertainty estimation includes the following steps: Figure 1 S1, generating a hash code set Assigning a binary hash code to each user Target encryption key as identity information And construct a key space: Generating a hash code set The specific working steps are as follows: S1.1, the hash code set Includes a plurality of binary hash codes ( , The number of a plurality of binary hash codes), each binary hash code Each bit The value probability (0 or 1) of the first bit (the bit (the , Binary hash code length) of the binary hash code ) is the same, and the different bits in each binary hash code are independent, so as to ensure that the binary hash code is biased in the binary space - there is no case that a bit in the binary hash code is long-term 0 or long-term 1, which fundamentally avoids the collision risk (two different user hash codes are highly similar) caused by the bias in the binary hash code S1.2, preset distance threshold The distance threshold is a quality control parameter for ensuring that the minimum Hamming distance of all code pairs in the hash code set meets the minimum difference requirement of the application, and the specific value depends on the requirements of the application, such as a smaller distance threshold for a shorter hash code length or a lower requirement for distinguishability, and a larger distance threshold for a high requirement for distinguishability and strong fault tolerance), the Hamming distance between any two binary hash codes and binary hash codes ( ) is greater than the distance threshold ; , so as to ensure that there is clear distinguishability between multiple binary hash codes in the hash code set ; The generation of the hash code set can be adopted by any method capable of generating a set of binary vectors uniformly distributed in binary space and having a sufficient distance from each other, such as generating a hash code set based on random projection, PCA dimensionality reduction and quantization, or a more complex optimization algorithm ; S1.3, after receiving the user registration request, and the registration request includes identity information (such as user ID, name, etc.), a unique, unoccupied binary hash code from the hash code set is assigned to each user hash code set as the target encryption key of the identity information , and the assigned binary hash code is permanent and unique.

[0032] The working principle of constructing the key space in S1 is as follows: a large-scale face data set (public data sets such as LFW (Labeled Faces in the Wild), AgeDB-30, and CFP (CelebFaces Attributes) are selected for training and evaluation) is used to pre-train a basic feature encoder (such as ArcFace, MagFace, etc.), and multiple (for example, 5-10) face images (each face image is a face image under different angles, expressions, and illuminations) are collected by a camera or other biometric feature collection device (for example, a high-resolution camera). The basic feature encoder extracts the original deep face features of each face image ; Step two: establish a mapping relationship between the registration request, the corresponding target encryption key and the face features , and the registration request, the target encryption key and human face features is constructed as a key space; By establishing a mapping relationship, a highly irreversible conversion is realized, which means that even if an attacker obtains the final target encryption key , it is impossible to deduce the human face features from it , thereby providing industry-leading privacy protection.

[0033] The pre-training of the basic feature encoder works as follows: perceive a large-scale face dataset containing a large number of face images of different identities, poses, expressions, and lighting conditions; and align the face images in the large-scale face dataset, and then standardize them to a uniform size and pixel range; A deep neural network architecture is adopted to extract face features layer by layer through a multi-layer network structure: the deep neural network architecture includes a bottom network, a middle network, and a high network. The bottom network is composed of multiple convolutional layers for learning the basic visual features of face images; the middle network integrates the basic visual features into more complex local features through the combination of convolutional layers and pooling layers; the high network further refines the face features directly related to identity; and through a nonlinear activation function, the expression ability of the face features is enhanced, and through a residual connection technology, the problem of gradient disappearance during training of the deep neural network architecture to learn the basic feature encoder is solved.

[0034] As Figure 2 , when constructing the key space, the basic feature encoder in step one, specifically adopts an industry-leading pre-trained basic feature encoder as the backbone, such as ArcFace based on ResNet50, and other models such as MagFace, AdaFace, PartialFace, DCTDP, etc.; the feature dimension output by these models is usually 512 dimensions; and selects public datasets such as LFW (Labeled Faces in the Wild), AgeDB-30, and CFP (CelebFacesAttributes) to train and evaluate the basic feature encoder , in order to simulate an open set scenario, additional face images from the CelebA dataset are introduced as "unknown" interference samples; ensure that the ratio of known and unknown samples in each face dataset meets the actual open set scenario (e.g., 95.24% open set ratio); and the pre-trained basic feature encoder Besides models like ArcFace and MagFace, other advanced face recognition backbone networks, such as MobileFaceNet and EfficientNet, can be used, or feature extractors pre-trained using self-supervised learning and contrastive learning methods can be combined; only the basic feature encoder is required. Capable of extracting facial features with sufficient discriminative power That's all.

[0035] To enable the basic feature encoder Able to adapt to facial features Mapped to the target encryption key assigned to the user S2. Use Low-Rank Adaptation (LoRA) parameters for the feature encoder. Fine-tuning yields a personalized feature encoder ,like Figure 3 The specific working principle is as follows: S2.1 Select face images (multiple face images with different conditions) corresponding to the same identity information in the key space as query samples. To start training, first calculate the total loss. (Total loss) For feature space loss + Hash space loss +Evidence Classification Loss Two low-rank matrices are inserted into the fully connected layer near the output of the basic feature encoder (referring to the layer in the network that is most sensitive to and discriminative of feature representation, such as the fully connected layer near the output, which directly determines the dimension of the final feature). and This forms a new weight matrix, resulting in the personalized feature encoder after initial fine-tuning. Personalized feature encoder Extract adaptive features from the query sample: low-rank matrix and Low- and medium-rank matrices The dimension is Used to map high-dimensional features to a low-dimensional space; low-rank matrix The dimension is This is used to map low-dimensional features back to the original dimension and is superimposed on the output of the fully connected layer near the output of the basic feature encoder. Facial feature dimension (basic feature encoder) Output facial features (dimensions) A low-rank matrix and The low-rank parameter r is critical, and is usually much smaller than the base feature encoder The dimension of the original matrix in the base feature encoder: for example, for a 512-dimensional feature, the low-rank parameter r can be set to 4, 8, 16, etc. smaller value, ensuring the lightweight of fine-tuning; S2.2, input adaptive features to the mapping function, which includes a flip layer, a dropout layer, a fully connected layer and a Softplus activation function, and outputs the original evidence value and the predicted key through the fully connected layer branch, and then processes the original evidence value through the Softplus activation function to ensure that the output is non-negative, and finally obtains the evidence value, and synchronously generates the predicted key and the evidence value; S2.3, calculate the fine-tuned base feature encoder The total loss after the base feature encoder: feature space loss + hash space loss + evidence classification loss; The feature space loss is the cosine similarity loss between the adaptive feature and the face feature; the hash space loss is the matching degree between the predicted key and the target encryption key; the evidence classification loss includes the mean square error term and the KL divergence regularization term; the mean square error term is the probability value of the adaptive feature corresponding to the face image belonging to different users, and the KL divergence regularization term is the difference between the probability value of the adaptive feature corresponding to the face image and the actual user probability value of the face image; S2.4, preset loss threshold, if the total loss is less than or equal to the preset loss threshold, it is judged that the base feature encoder still needs to be adjusted, and back propagation is performed: the personalized feature encoder The output of the mapping function in the base feature encoder Close to the output of the fully connected layer, calculate the gradient of the predicted key and the evidence value on the total loss: based on the hash space loss and the evidence classification loss, calculate the partial derivative of the loss with respect to the mapping function; then calculate the gradient of the two inserted low-rank matrices: based on the feature space loss and the hash space loss, calculate the partial derivative of the loss with respect to the low-rank matrix; S2.5, using an optimizer to update the weight values of the two inserted low-rank matrices according to the back propagation gradient based on the gradient descent principle: the new weight value is equal to the old weight value minus the learning rate multiplied by the gradient; S2.6, iterative process: repeat S2.1-S2.5 for each training iteration until the total loss is greater than the preset loss threshold, and obtain the fine-tuned personalized feature encoder of each query sample .

[0036] The personalized feature encoder Fine-tuning alternatives: Besides the standard Low-Rank Adaptation (LoRA) parameter structure, other efficient fine-tuning techniques can be explored, such as Prefix-Tuning, Prompt-Tuning, or Adapter-based methods, which are also designed to efficiently adapt large pre-trained personalized feature encoders with a small number of parameters .

[0037] As Figure 4 , S3, the authentication phase receives the authentication request, calls out the input identity information in the authentication request, compares whether there is the same identity information as the output identity information in the key space, if there is identity information matching the output identity information in the key space, then the real-time face image collected by the camera or other biometric feature acquisition device (for example, high-resolution camera) is collected, and the adaptive feature is extracted by the corresponding personalized feature encoder ; ; And input the adaptive feature to the mapping function (The network structure of the mapping function includes flip layer, dropout layer, fully connected layer and Softplus activation function, and the network structure can be adjusted according to actual needs, such as increasing or reducing the number of layers, using different activation functions, etc., to optimize performance), the mapping function Output the predicted key and the evidence value e, the corresponding steps are: Step one: receive the adaptive feature (Where is the feature dimension of the adaptive feature , such as 512 dimensions), the mapping function Each layer operates as follows: the flip layer randomly rearranges the feature dimension of the adaptive feature , and outputs the flipped feature: Where is a random dimension flip operation (such as dimension exchange by random permutation matrix); The Dropout layer randomly discards part of the dimensions in the adaptive feature with a probability (such as 0.5), and outputs: Where is the Dropout operation, and the discarded dimension value is set to 0; Step two: fully connected layer combination (feature compression and branch): The first fully connected layer (ReLU activation): compresses the feature from the feature dimension compressed to dimension (e.g., 256 dimension): where is the weight matrix, is the bias; second fully connected layer (two branches): key generation branch: compressed to hash code length dimension (e.g., 128 dimension), output continuous value vector : , where , is the key branch parameter; evidence value generation branch: output 1-dimensional original evidence value , , is the evidence branch parameter; Step three: predict the key and the final expression of the evidence value e: predicted key (discretization): hard quantization is performed on the continuous vector to obtain 0 / 1 discrete hash code : where is the kth dimension of , and the threshold value 0.5 ensures that the discretization result is unique; evidence value (Softplus activation): Softplus activation is applied to the original evidence value to ensure that the output is non-negative: .

[0038] receive the mapping function output the predicted key and the evidence value e; the evidence deep learning (based on Dirichlet distribution) is used to convert the evidence value e into uncertainty where is the dummy class weight, which is used to enhance the modeling ability for unknown classes, is the cumulative evidence (including prior knowledge) for "the input belongs to the jth class"; preset uncertainty threshold , compare the uncertainty with the uncertainty threshold , and the preset uncertainty threshold The working principle is as follows: the perception verification set (its source is similar to the face data set, such as the public data sets mentioned above, LFW, AgeDB-30, CFP, etc., or specific face data collected according to the actual application scene) determines the uncertainty threshold by analyzing the false acceptance rate (the probability of misjudging an unregistered user or a malicious attacker as a legitimate user) and the false rejection rate (the probability of misjudging a legitimate registered user as an unknown user and rejecting it) curve on the verification set , traverse the false acceptance rate and false rejection rate corresponding to the candidate threshold, find the balance point on the curve or select the candidate threshold as the uncertainty threshold according to the requirement to balance the security and usability, for example, the uncertainty threshold = 0.15; Compared with the traditional open set recognition based on similarity threshold, the present application provides a more fine and reliable unknown sample discrimination mechanism, which significantly enhances the security and practicality of the system in a real open environment.

[0039] and according to the authentication request, the corresponding target encryption key is called from the key space , and the target encryption key is compared with the predicted key , and the user identity is judged through two different stages: Stage one: uncertainty filtering: if the uncertainty is greater than the uncertainty threshold , it indicates that the personalized feature encoder is extremely uncertain about the identity attribution of the current face image (the face image collected when the user inputs information (before authentication)), and this situation usually occurs in the following situations: The face does not belong to any registered user (unknown user), the face is fake (live attack), the face image quality is extremely poor, so that the personalized feature encoder cannot effectively identify; At the same time, it is judged as authentication failure and access is refused, thereby effectively preventing unknown or malicious users from entering the subsequent precise matching link; If the uncertainty is less than or equal to the uncertainty threshold , it indicates that the personalized feature encoder has high enough confidence in the identity attribution of the current face image, and it is judged that the authentication request may be a legitimate and registered user authentication request, and then enters the next stage of precise key matching; Stage two: precise key matching: prerequisite: only execute after the first stage of uncertainty filtering passes; judgment logic: Adaptive features (the face image collected after the authentication request is input is passed through the personalized feature encoder extracted adaptive feature ) corresponding target encryption key with personalized feature encoder output predicted key bit-by-bit comparison if the target encryption key is equal to the predicted key , it means that the target encryption key is completely matched with the predicted key , and the authentication is determined to be successful if the target encryption key is not equal to the predicted key ( even only one bit is different ) : it means that the target encryption key is not completely matched with the predicted key , and the authentication is determined to be failed

[0040] The above shows and describes the basic principles, main features and advantages of the present application. It should be understood by those skilled in the art that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only preferred examples of the present application, and are not intended to limit the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A privacy-preserving open-set face recognition method based on Lora and uncertainty estimation, characterized in that, The method comprises the following steps: S1, generating a hash code set, pre-training a basic feature encoder to extract face features, mapping the face features to a unique binary hash code in the hash code set, defining the target encryption key, decoupling the potential correlation between the face features and the target encryption key, and constructing a key space; S2, selecting a face image in the key space as a query sample, using a low-rank adaptive technology to fine-tune the basic feature encoder into a personalized feature encoder, and converging the face features of the query sample under different conditions to the mapped target encryption key; S3, when fine-tuning the basic feature encoder, jointly learning the evidence value about the output, and deducing the degree of "uncertainty" when the personalized feature encoder outputs the predicted key; in the authentication stage, the uncertainty is calculated, if the uncertainty value is greater than the preset uncertainty threshold, the access request is rejected, if the uncertainty value is not greater than the uncertainty threshold, the predicted key is matched with the encryption key in the database bit by bit.

2. The privacy-preserving open-set face recognition method based on Lora and uncertainty estimation according to claim 1, characterized in that: The hash code set includes a plurality of binary hash codes, the probability of each bit in each binary hash code is the same, and the different bits in each binary hash code are correlated and independent; and a distance threshold is preset, the Hamming distance between any two binary hash codes in the hash code set is greater than the distance threshold, and the binary hash code allocated for each user is an unoccupied binary hash code.

3. The privacy-preserving open-set face recognition method based on Lora and uncertainty estimation according to claim 2, characterized in that: The basic feature encoder is pre-trained: perceiving a large-scale face dataset, the large-scale face dataset contains a large number of face images under different identities, different poses, expressions and illumination conditions; and aligning the face images in the large-scale face dataset, and then standardizing them to a uniform size and pixel range; A deep neural network architecture is used to extract face features layer by layer through a multi-layer network structure: the deep neural network architecture includes a bottom network, a middle network and a high network; The bottom network is composed of multiple convolutional layers, which is used to learn the basic visual features of the face image; The middle network integrates the basic visual features into more complex local features through the combination of convolutional layers and pooling layers; The high network further refines the face features directly related to the identity; and enhances the expression ability of the face features through a nonlinear activation function, and solves the problem of gradient disappearance when training the basic feature encoder through residual connection technology.

4. The privacy-preserving open-set face recognition method based on Lora and uncertainty estimation according to claim 2, characterized in that: The key space is constructed: after perceiving the user registration request, multiple face images corresponding to the user are collected, the basic feature encoder extracts the face features of each face image, the unallocated binary hash code in the hash code set is called out, and is mapped to the face features, the registration request, the target encryption key and the face features are associated to construct the key space.

5. The privacy-preserving open-set face recognition method based on Lora and uncertainty estimation according to claim 1, characterized in that: The working steps of the low-rank adaptive technology for fine-tuning the basic feature encoder into a personalized feature encoder are as follows: S2.1, selecting the face images corresponding to the same identity information in the key space as query samples, and starting training, inserting two low-rank matrices in the full connection layer close to the output of the basic feature encoder to form a new weight matrix, obtaining the personalized feature encoder after the first fine-tuning, and the personalized feature encoder extracts adaptive features in the query sample; S2.2, input adaptive features to the mapping function, the mapping function includes a flip layer, a dropout layer, a fully connected layer and a Softplus activation function, the original evidence value and the predicted key are output through the fully connected layer branch, and the original evidence value is processed through the Softplus activation function to ensure that the output is a non-negative value, and the final evidence value is obtained, and the predicted key and the evidence value are generated synchronously; S2.3, calculate the total loss after fine-tuning the basic feature encoder: the sum of the feature space loss, the hash space loss and the evidence classification loss; The feature space loss is the cosine similarity loss between the adaptive features and the face features; the hash space loss is the matching degree between the predicted key and the target encryption key; the evidence classification loss includes a mean square error term and a KL divergence regularization term; the mean square error term is the probability value of the adaptive features corresponding to the face image belonging to different users, and the KL divergence regularization term is the difference between the probability value of the adaptive features corresponding to the face image and the probability value of the actual user to which the face image belongs; S2.4, preset loss threshold, if the total loss is less than or equal to the preset loss threshold, it is judged that the basic feature encoder still needs to be adjusted, and back propagation is performed: the output of the mapping function in the personalized feature encoder is reversely transmitted to the fully connected layer close to the output of the basic feature encoder, and the gradient of the predicted key and the evidence value to the total loss is calculated: based on the hash space loss and the evidence classification loss, the partial derivative of the loss to the mapping function is calculated; Then calculate the gradient of the inserted two low-rank matrices: based on the feature space loss and the hash space loss, the partial derivative of the loss to the low-rank matrix is calculated; S2.5, using the optimizer to update the weight value of the initial inserted two low-rank matrices according to the back propagation gradient according to the gradient descent principle: the weight new value is equal to the weight old value minus the learning rate multiplied by the gradient; S2.6, iterative process: repeat S2.1-S2.5 in each training iteration until the total loss is greater than the preset loss threshold, and obtain the fine-tuned personalized feature encoder of each query sample.

6. The privacy-preserving open-set face recognition method based on Lora and uncertainty estimation according to claim 5, characterized in that: The authentication stage receives an authentication request, calls out the input identity information in the authentication request, compares whether there is identity information with the same identity information as the output identity information in the key space, if there is identity information matching the input identity information in the key space, then through the real-time collected face image, then call out the personalized feature encoder corresponding to the identity information, extract the adaptive features through the corresponding personalized feature encoder; And input the adaptive features into the mapping function, the mapping function outputs the predicted key and the evidence value.

7. The privacy-preserving open-set face recognition method based on Lora and uncertainty estimation according to claim 6, characterized in that: When the personalized feature encoder extracts the adaptive features, sum all the intermediate evidence values pointing to the same known user to form the total positive class evidence of the known user; all the intermediate evidence values not pointing to the known user are summarized to form the total evidence of the dummy class, reflecting the support degree of the unknown user: the uncertainty is converted by using the dummy class weight divided by the sum of all input total positive class evidence belonging to each class.

8. The Lora and uncertainty estimation based privacy-preserving open-set face recognition method according to claim 1, characterized in that: The authentication stage: preset uncertainty threshold; If the uncertainty is less than or equal to an uncertainty threshold, it is determined that the authentication request is successful, and if the uncertainty is greater than the uncertainty threshold, it is determined that the authentication request fails; after the authentication request is successful, the target encryption key is compared with the predicted key; If the target encryption key is equal to the predicted key, it is determined that the authentication is successful; if the target encryption key is not equal to the predicted key, it is determined that the authentication fails.

9. The privacy-preserving open-set face recognition method based on Lora and uncertainty estimation according to claim 8, characterized in that: The preset uncertainty threshold comprises the following steps: The verification set is perceived by analyzing the false acceptance rate and the false rejection rate curves on the verification set, wherein the false acceptance rate is the probability of misjudging an unregistered user or a malicious attacker as a legal user, and the false rejection rate is the probability of misjudging a legal registered user as an unknown user and rejecting it, the false acceptance rate and the false rejection rate corresponding to the candidate threshold are traversed, and the equilibrium point on the curve is found or the candidate threshold is selected as the uncertainty threshold according to the requirement.

10. The privacy protection open set face recognition method based on Lora and uncertainty estimation according to claim 9, wherein the comparison between the target encryption key and the predicted key is a complete consistency comparison bit by bit, rather than a similarity comparison.

Images