Face recognition method and device for protecting visual privacy, server and user terminal

Abstract

The application discloses a face recognition method and device for protecting visual privacy, a server and a user terminal, and relates to the field of image processing.The technical scheme is as follows: receiving a visual anonymous image sent by a user terminal and a feature map extracted by a feature extraction model deployed on the user terminal, wherein the feature map is obtained by extracting a face difference image by the feature extraction model; inputting the visual anonymous image, the feature map and weight parameters output by the feature extraction model in a training stage into a pre-trained face recognition model embedded with a feature fusion module for feature fusion and face recognition; and when the weight parameters are correct, outputting a face identity feature vector from the face recognition model to the user terminal.The application not only ensures that the visual anonymous image is anonymous to human eyes, but also improves the identity recognition accuracy of the visual anonymous image, thereby solving the balance problem between visual privacy protection and recognition accuracy of the anonymous image.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to a method, apparatus, server, and client for facial recognition that protects visual privacy. Background Technology

[0002] Facial recognition, as an important biometric identification technology, has been widely used in many fields such as military, finance, and daily life. However, while enjoying the convenience brought by facial recognition, people also face the risk of their personal privacy information being leaked. This is because the rich personal information contained in facial images may be leaked or misused during the facial recognition process, and since facial images are a visual medium that can directly expose personal identity information, once leaked, they cannot be recovered.

[0003] Existing technologies do not consider the visual usability of privacy-preserving images. For example, many applications rely on automatic facial recognition to ensure secure service functionality, while also needing to consider the privacy of individuals captured by the camera. Addressing the trade-off between privacy and utility, existing technologies have proposed a novel framework, PRO-Face, for privacy preservation and recognizable obfuscation of facial images. This framework supports the use of various obfuscation methods to anonymize facial images. Then, a Siamese neural network is used to fuse the original image with its obfuscated form, resulting in a privacy-preserving image that is visually similar to the obfuscated image but can be recognized by a machine as the original identity. However, the recognition accuracy of this method on anonymized images needs improvement.

[0004] Therefore, there is a need for a face recognition method that protects visual privacy, that is, it can protect visual privacy information in face images while ensuring the accuracy of face recognition. Summary of the Invention

[0005] To address the shortcomings of existing privacy-preserving face recognition methods, this invention provides a face recognition method, device, server, and user terminal that protect visual privacy. The invention receives visually anonymized images uploaded from the user terminal on the server side, making the original face images unrecognizable to the human eye, thus solving the problem of facial privacy information leakage. To improve the recognition accuracy of anonymized images, the invention receives feature maps extracted from face difference images by a feature extraction model deployed on the user terminal. Then, using a pre-trained face recognition model embedded with a feature fusion module, the feature maps and weight parameters are fused into the face recognition model during the extraction of facial identity feature vectors from the visually anonymized images. The resulting facial identity feature vector can be used for accurate identity recognition. Furthermore, to ensure the security of face recognition, correct weight parameters must be used during the recognition stage of the face recognition model to output correct face recognition results; otherwise, the recognition result will be incorrect. In summary, this invention not only ensures the anonymity of visually anonymized images to the human eye but also improves the accuracy of identity recognition of visually anonymized images, thereby solving the problem of balancing visual privacy protection and recognition accuracy of anonymized images.

[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0007] A first aspect of this application provides a face recognition method for protecting visual privacy, applied to a server, the method comprising:

[0008] The system receives a visually anonymized image sent by a user terminal and a feature map extracted by a feature extraction model deployed on the user terminal, wherein the feature map is obtained by extracting a face difference image through the feature extraction model.

[0009] The visual anonymized image, feature map, and weight parameters output by the feature extraction model during the training phase are input into a pre-trained face recognition model embedded with a feature fusion module for feature fusion and face recognition. When the weight parameters are correct, the face recognition model outputs a face identity feature vector and sends it to the user terminal.

[0010] In one implementation, the feature extraction model is trained on the server, and the trained feature extraction model is distributed to at least one user terminal to complete the deployment of the feature extraction model.

[0011] In one implementation, the feature extraction model is modified based on the MobileFaceNet network. Specifically, the output layer of the MobileFaceNet network is deleted, and a feature map extraction module and a weight module are inserted at different positions in the MobileFaceNet network. The feature map extraction module consists of an upsampling layer and a convolutional layer, and the weight module consists of a vector matrix. The model parameters of the feature map extraction module are set according to the face recognition model.

[0012] During the training phase, the feature extraction model outputs the feature map and weight parameters; during the recognition phase, the feature extraction model only outputs the feature map, while the weight parameters serve as the key for the face recognition model to perform recognition.

[0013] In one implementation, an ArcFace loss function and an L2 loss function are used to construct a total loss function, and the model parameters of the feature extraction model are updated based on the total loss function.

[0014] In one implementation, the fusion formula for combining the visual anonymity image, feature map, and weight parameters is as follows: Where I represents the original face image, This represents the visually anonymized image obtained after visual anonymization preprocessing of the original face image I. Represents a face difference image; M i (I) represents the feature map output by the feature extraction model M at the end of the i-th stage; ω represents the feature map output by the face recognition model F at the end of stage i. i represents the weight parameters, and sigmoid represents the activation function.

[0015] In one implementation scheme, applied to the user end, the method also includes:

[0016] The user's original face image is detected and captured, and the original face image is preprocessed to anonymize visual information to obtain the visually anonymized image;

[0017] The face difference image is obtained by subtracting the visual anonymization image from the original face image pixel by pixel.

[0018] At least one set of feature maps is extracted from the face difference image according to the feature extraction model;

[0019] Send the visual anonymization image and feature map to the server;

[0020] The system receives a facial identity feature vector sent by the server and performs facial recognition and verification based on the facial identity feature vector. The facial identity feature vector is obtained by performing feature fusion and facial recognition on the visual anonymized image, feature map, and weight parameters output by the feature extraction model during the training phase using a pre-trained facial recognition model with an embedded feature fusion module. The facial recognition model outputs the model when the weight parameters are correct.

[0021] In one implementation, the anonymization preprocessing includes Gaussian blur, median filtering, pixelation, and hybrid blur; wherein the hybrid blur involves dividing an original face image into multiple image blocks, and performing Gaussian blur, median filtering, and pixelation on each image block.

[0022] A second aspect of this application provides a face recognition device for protecting visual privacy, applied to a server, comprising:

[0023] The receiving module is used to receive a visually anonymous image sent by the user terminal and a feature map extracted by the feature extraction model deployed on the user terminal, wherein the feature map is obtained by extracting a face difference image through the feature extraction model.

[0024] The fusion recognition module is used to input the visual anonymized image, feature map, and weight parameters output by the feature extraction model during the training phase into a pre-trained face recognition model that is embedded with a feature fusion module for feature fusion and face recognition. When the weight parameters are correct, the face recognition model outputs a face identity feature vector and sends it to the user terminal.

[0025] A third aspect of this application also provides a server comprising a processor and a memory, the memory storing at least one instruction loaded and executed by the processor to perform the operations of a face recognition method for protecting visual privacy as described in the first aspect of this application.

[0026] A fourth aspect of this application also provides a user terminal, the user terminal including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to perform the operations performed by a face recognition method for protecting visual privacy as described in the first aspect of this application.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. This invention receives visually anonymized images uploaded from user terminals on the server side, making the original face images unrecognizable to the human eye, thus solving the problem of facial privacy leakage. To improve the recognition accuracy of anonymized images, this invention receives feature maps extracted from face difference images by a feature extraction model deployed on the user terminal. Then, using a pre-trained face recognition model embedded with a feature fusion module, the feature maps and weight parameters are fused into the face recognition model during the extraction of facial identity feature vectors from the visually anonymized images. The resulting facial identity feature vector can be used for accurate identity recognition. Furthermore, to ensure the security of face recognition, correct weight parameters must be used during the recognition stage of the face recognition model to output correct face recognition results; otherwise, the recognition results will be incorrect. In summary, this invention not only ensures the anonymity of visually anonymized images to the human eye but also improves the accuracy of identity recognition of visually anonymized images, thereby solving the balance problem between visual privacy protection and recognition accuracy of anonymized images.

[0029] 2. The server of this invention only receives the visually anonymized image after anonymization preprocessing and the feature map extracted by the feature extraction model, and does not receive the original face image obtained by the user, thereby protecting the user's visual privacy.

[0030] 3. This invention only requires training a lightweight feature extraction model on the server. Training a lightweight feature extraction model can effectively reduce the server's operating resources. Then, it is distributed to the user terminal, where the lightweight feature extraction model is deployed to extract feature maps from the face difference images generated by the user terminal. The feature maps can effectively improve the accuracy of identity recognition of anonymous images. The weight parameters output by the feature extraction model during the training phase can protect the security of face recognition. Furthermore, the input of the feature extraction model is the face difference image, which makes the feature extraction model pay more attention to the information in the face difference image, thereby more effectively extracting feature maps for assisting face recognition.

[0031] 4. This invention uses anonymization preprocessing operations (including blurring, pixelation, etc.) to make the original face image unrecognizable by human vision, thereby protecting the visual privacy information in the original face image. The resulting privacy-protected image can retain the effectiveness of face recognition and monitoring. Attached Figure Description

[0032] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0033] Figure 1 A flowchart illustrating a face recognition method for protecting visual privacy provided by an embodiment of the present invention is shown.

[0034] Figure 2 This diagram illustrates the structure of a face recognition system that implements visual privacy protection according to an embodiment of the present invention.

[0035] Figure 3 A schematic diagram illustrating the process of implementing visual privacy protection for face recognition provided by an embodiment of the present invention is shown;

[0036] Figure 4 This diagram illustrates the structure of a face recognition device for protecting visual privacy according to an embodiment of the present invention.

[0037] Figure 5 A schematic diagram of the structure of a server provided in an embodiment of the present invention is shown. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0039] It should be noted that the terms "comprising" or "may include" used in the various embodiments of this application indicate the presence of the claimed function, operation, or element, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in the various embodiments of this application, the terms "comprising," "having," and their cognates are intended only to indicate a specific feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or adding one or more combinations of the foregoing.

[0040] Currently, facial recognition methods that protect user privacy can be broadly categorized into the following three types:

[0041] 1) Cryptographic-based privacy-preserving face recognition methods

[0042] In this type of work, facial recognition is primarily performed in the cryptographic domain. For example, feature extraction and similarity calculation need to be done directly on the encrypted image or through the execution of certain security protocols. One example is a biometric identification system based on visual cryptography, which protects biometric privacy by storing biometrics in a separate database. Visual cryptography combines perfect cryptography and shared keys with images, thereby eliminating the complex operations found in existing cryptography-based privacy protection schemes.

[0043] Encryption-based methods have minimal impact on recognition accuracy because almost all operations are lossless. Furthermore, the computational security of the encryption techniques employed strongly guarantees their effectiveness, often requiring attackers to solve a difficult and complex computational problem. However, these methods may be limited in practical use due to their prohibitively high computational and communication costs. Moreover, these methods have low versatility because most are tightly coupled to very specific face recognition schemes, requiring different privacy protection methods to be designed for different face recognition approaches.

[0044] 2) Privacy-preserving face recognition method based on frequency domain learning

[0045] Frequency domain learning is frequently used in image compression. Compression removes redundant information while retaining information meaningful for the comprehension task, providing a novel approach for various image processing tasks. For example, applying frequency domain learning to model training involves using Discrete Cosine Transform (DCT) to convert images to the frequency domain, and then modifying the model's input layer to accommodate frequency-domain images of different sizes. Experimental results show that these methods can improve the accuracy of task models and accelerate training.

[0046] This paper addresses the contribution of frequency domain channels to visual visualization and face recognition. For example, existing techniques propose an analysis network to identify channels that are important for image visualization but not crucial for maintaining high recognition accuracy. Then, a fast masking method is proposed to protect the remaining frequency domain channels. Experimental results show that this scheme achieves almost no decrease in face recognition accuracy while maintaining privacy. Existing techniques also propose a novel privacy-preserving face recognition method employing collaborative reasoning in the frequency domain. This method fully leverages the advantages of a client-server model, allowing the client to send non-critical channels of the frequency domain image and identity feature masks to the server. The server then receives this information and performs face recognition. Experimental results show that compared to the unprotected face recognition algorithm ArcFace, this method achieves the same recognition accuracy and computational cost without the risk of privacy leakage.

[0047] While frequency domain learning-based methods can maintain face recognition accuracy, they require modifications to the model structure for different face recognition models. Typically, the input layer of the model needs to be modified to adapt to multi-channel frequency domain images. The modified model then needs to be retrained on large datasets, which consumes significant time and computational resources.

[0048] 3) Other privacy-preserving facial recognition methods

[0049] This category of work utilizes various techniques to protect the privacy of face recognition systems, such as image perturbation, federated learning, and clustering methods. Some studies leverage GAN technology to perturb images for privacy protection. For instance, existing techniques address the vulnerability of data to attacks during face recognition model training by proposing a general privacy protection method. This method uses data generated by a Bayesian GAN and noisy labels to train the model, preventing attackers from correctly predicting the training data and thus improving model security. A highly efficient privacy-preserving face recognition scheme based on clustering methods is also proposed to address the exorbitant cost of encrypted face image queries. Specifically, a clustering-based multiple matching scheme can effectively improve face recognition accuracy without increasing time complexity.

[0050] The aforementioned three categories of existing technologies do not consider the visual usability of privacy-preserving images. For example, many applications developed based on these methods rely on automatic facial recognition to ensure secure service functions, while also needing to consider personal privacy under the camera's view. Addressing the trade-off between privacy and utility, existing technologies have proposed a novel framework, PRO-Face, for privacy preservation and recognizable obfuscation of facial images. This framework supports the use of various obfuscation methods to anonymize facial images. Then, a Siamese neural network is used to fuse the original image with its obfuscated form, resulting in a privacy-preserving image that is visually similar to the obfuscated image but can be recognized by a machine as the original identity. However, the recognition accuracy of this method on anonymized images needs improvement.

[0051] Therefore, a face recognition method that protects visual privacy is needed, ensuring both the preservation of visual privacy information in face images and the accuracy of face recognition. This embodiment provides a face recognition method that protects visual privacy. This method is applied to a server, receiving visually anonymized images uploaded from user terminals. This prevents the original face images from being effectively recognized by the human eye, thus solving the problem of facial privacy information leakage. To improve the recognition accuracy of anonymized images, this invention extracts feature maps from the face difference image based on a feature extraction model deployed on the user terminal. These feature maps are then fused into the face recognition model during the extraction of facial identity feature vectors from the visually anonymized image. The resulting identity features can be used for accurate face identification. Furthermore, to ensure the security of face recognition, correct weight parameters must be used during the recognition stage of the face recognition model to obtain correct face recognition results; otherwise, errors will occur. In summary, this invention not only ensures the anonymity of visually anonymized images to the human eye but also improves the accuracy of identity recognition for visually anonymized images, thereby solving the balance problem between visual privacy protection and recognition accuracy of anonymized images.

[0052] Understandably, the face recognition method provided in this embodiment is applicable to edge-cloud collaborative scenarios such as video surveillance and "face-scanning" payment. The lightweight feature extraction model has a small memory footprint and can be deployed on smart terminal devices such as cameras and mobile phones. The terminal device sends anonymized images and feature maps to the cloud. The cloud server deploys a face recognition model and a feature map fusion function. The cloud server only receives the anonymized visually anonymized image and the feature map extracted by the feature extraction model deployed on the terminal device, and does not receive the original face image, thereby protecting the user's visual privacy.

[0053] Please refer to Figure 1 , Figure 1 The diagram illustrates a flow chart of a face recognition method for protecting visual privacy provided by an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0054] S110, receiving a visually anonymous image sent by the user terminal and a feature map extracted by a feature extraction model deployed on the user terminal, wherein the feature map is obtained by extracting a face difference image through the feature extraction model.

[0055] In this embodiment, the user terminal includes, but is not limited to, smart terminal devices that require facial recognition authentication, such as mobile phones, cameras, computers, and iPads.

[0056] Feature maps can assist in facial recognition using anonymous images, but they cannot be effectively used for facial recognition and verification. Feature maps are highly abstract, and if they are accidentally leaked, attackers will not be able to obtain any valid identity information from them.

[0057] Visually anonymized images cannot be effectively recognized by the human eye, thus protecting the privacy of human vision; in addition, the recognition accuracy of visually anonymized images is low, thus protecting the privacy of machine vision.

[0058] S120, the visual anonymized image, feature map, and weight parameters output by the feature extraction model during the training phase are input into a pre-trained face recognition model embedded with a feature fusion module for feature fusion and face recognition. When the weight parameters are correct, the face recognition model outputs a face identity feature vector and sends it to the user terminal.

[0059] Since the feature map described in step S110 can only assist in face recognition of anonymous images and cannot be effectively used for face recognition and verification, in order to improve the recognition accuracy of visually anonymous images, this embodiment provides the weight parameters of the feature extraction model during the training phase as an authentication key. When the weight parameters are correct, the security of face recognition can be improved.

[0060] In this embodiment, the face recognition model used includes, but is not limited to, commonly used face recognition models such as AdaFace, ArcFace, or CosFace. This embodiment relies only on a pre-trained high-precision face recognition model, without requiring training of that model.

[0061] The facial recognition method described in this embodiment is applicable to edge-cloud collaborative scenarios such as video surveillance and "face-scanning" payment. Specific implementation examples are provided below. Figure 3 As shown, Figure 3 This illustration shows a flowchart of a face recognition method provided by an embodiment of the present invention, performing face recognition in an edge-cloud collaborative application scenario. First, a trained feature extraction model is sent to the user terminal, such as a smart camera or smartphone. After the user terminal captures the image, it undergoes face detection to obtain the original face image. Then, an anonymization preprocessing method is used to obtain a visually anonymized image. The visually anonymized image is subtracted from the original face image to obtain a face difference image. Finally, the face difference image is fed into the feature extraction model to obtain a feature map. The user terminal sends the visually anonymized image and the feature map to the server. Base64 encoding can be used to transmit the visually anonymized image to save bandwidth. The received facial anonymized image can be used for screen display or storage backup. Since the identity information of the facial anonymized image cannot be perceived by the human eye, this method achieves the effect of protecting the visual privacy information of the face image. In addition, if there is a need for facial recognition or verification, the anonymous facial image is input into a facial recognition model with added feature fusion function. At the same time, the correct weight parameters are used as a key to judge the recognition or verification (the weight parameters are obtained during the training phase of the feature extraction model). This will result in a correct facial recognition result. Otherwise, the result will be incorrect and the user will not be able to complete the identity recognition or verification.

[0062] In one embodiment, the feature extraction model is trained on the server, and the trained feature extraction model is distributed to at least one user terminal to complete the deployment of the feature extraction model.

[0063] In one embodiment, the feature extraction model is modified based on the MobileFaceNet network. Specifically, the output layer of the MobileFaceNet network is deleted, and a feature map extraction module and a weight module are inserted at different positions in the MobileFaceNet network. The feature map extraction module consists of an upsampling layer and a convolutional layer, and the weight module consists of a vector matrix. The model parameters of the feature map extraction module are set according to the face recognition model.

[0064] During the training phase, the feature extraction model outputs the feature map and weight parameters; during the recognition phase, the feature extraction model only outputs the feature map, while the weight parameters serve as the key for the face recognition model to perform recognition.

[0065] Specifically, this embodiment also provides the training process for the lightweight feature extraction model, which includes the following steps:

[0066] 1) Dataset and Preprocessing:

[0067] The MS1MV2 dataset is a large-scale face image dataset collected and released by Microsoft, containing 3.8 million face images from 85,000 identities; this dataset is used for training the model in this embodiment. The CelebA dataset contains 202,599 face images from 10,177 identities, and is labeled with approximately 40 facial attributes, such as whether the person is wearing glasses or smiling; the training set of this dataset is used for training the model in this embodiment, and the test set is used for testing the model. The LFW dataset contains 13,233 images of famous people worldwide from 5,749 identities, of which 1,680 people have two or more face images; this dataset provides a standardized face matching process for testing the model in this embodiment. The CALFW dataset is a cross-age dataset labeled based on the LFW dataset, designed for unconstrained face verification with large age differences, containing 12,176 face images from 4,025 identities; this dataset is used for testing the model in this embodiment. CPLFW dataset: A cross-pose dataset annotated based on the LFW dataset, containing 11,652 face images from 3,884 identities, with significant pose differences for each identity; this dataset was used for model testing in this embodiment. CFP-FP dataset: Contains 7,000 images from 500 identities, primarily collecting frontal and profile images of well-known figures; this dataset was used for model testing in this embodiment. AgeDB dataset: Contains 16,488 images from 568 celebrities, each image labeled with identity, age, and gender attributes. The average number of images per identity is 29, with the lowest and highest ages being 1 and 101, respectively; this dataset was used for model testing in this embodiment.

[0068] We used a pre-trained open-source face detection tool to detect, crop, and align the face images in the dataset above, keeping the face head in the center of the image, and set the image resolution to 112*112.

[0069] 2) Training a lightweight feature extraction model:

[0070] The lightweight feature extraction model was trained using the CelebA training set. Four anonymization preprocessing methods and the common face recognition model AdaFace were used during training, resulting in a single feature extraction model. The three anonymization preprocessing methods were: Gaussian blur (with a fixed blur kernel size of 31, using a blur kernel variance of 2-8 during training); median filtering (using blur kernel sizes of 7, 9, 11, 13, 15, 17, and 19 during training); pixelation (using pixel blocks of 4-10 pixels during training); and custom blending blur (dividing the image into 4, 9, 16, and 25 image blocks during training, with each block using a different blurring method). The entire training process was optimized using an SGD optimizer with a momentum of 0.0005 and a learning rate of 0.01.

[0071] Furthermore, in order to ensure that the feature extraction module can extract appropriate feature maps and weight parameters during training and improve the recognition accuracy of anonymous images, this embodiment uses the ArcFace loss function and the L2 loss function to construct a total loss function, and updates the model parameters of the feature extraction model according to the total loss function.

[0072] The total loss function used includes at least the loss between the facial identity features of the original face image and the facial identity features obtained by fusing feature maps and weight parameters.

[0073] Specifically, the total loss function used can be expressed as: in, Represents the total loss function; The ArcFace loss function is a commonly used loss function for training face recognition models. It guides the model to extract appropriate feature maps by evaluating the facial identity features of the fused feature maps. The L2 loss function measures the similarity between the facial identity features of the original image and the facial identity features of the fused feature map; λ1 and λ2 are respectively... The weight.

[0074] Furthermore, the L2 loss function Represented as: Where N represents the batch size set during model training, and I i This represents the i-th original face image. Indicates that image I i Anonymous images after visual anonymization preprocessing; E represents a pre-trained face recognition model, F represents a public face recognition model with added feature map fusion module, E(I i ) represents the facial features of the i-th original image. This represents the facial identity features of the fused feature map and weight parameters of the i-th anonymous image.

[0075] It should be understood that the facial identity feature vectors described in the above embodiments are all one-dimensional vectors, and the length of the facial identity feature vector can be 128, 256, 512, etc.

[0076] In one embodiment, the fusion formula for fusing the visual anonymization image, feature map, and weight parameters is as follows: Where I represents the original face image, This represents the visually anonymized image obtained after visual anonymization preprocessing of the original face image I. Represents a face difference image; M i (I) represents the feature map output by the feature extraction model M at the end of the i-th stage; ω represents the feature map output by the face recognition model F at the end of stage i. i represents the weight parameters, and sigmoid represents the activation function.

[0077] Since the face difference image is obtained by subtracting the original face image from the visually anonymized image, therefore,

[0078] In one embodiment, applied to a user terminal, the method further includes:

[0079] The user's original face image is detected and captured, and the original face image is preprocessed to anonymize visual information to obtain the visually anonymized image;

[0080] The face difference image is obtained by subtracting the visual anonymization image from the original face image pixel by pixel.

[0081] At least one set of feature maps is extracted from the face difference image according to the feature extraction model;

[0082] Send the visual anonymization image and feature map to the server;

[0083] The system receives a facial identity feature vector sent by the server and performs facial recognition and verification based on the facial identity feature vector. The facial identity feature vector is obtained by performing feature fusion and facial recognition on the visual anonymized image, feature map, and weight parameters output by the feature extraction model during the training phase using a pre-trained facial recognition model with an embedded feature fusion module. The facial recognition model outputs the model when the weight parameters are correct.

[0084] In this embodiment, the user terminal, such as a smart camera or smartphone, captures the user's image through a configured photosensitive element. After face detection, an original face image is obtained. Then, an anonymization method is used to obtain an anonymous face image. The anonymized image is subtracted from the original image to obtain a face difference image. Finally, the face difference image is fed into a feature extraction model to obtain a feature map. The user terminal sends the anonymized image and feature map to the cloud, and Base64 encoding is planned to be used for image transmission to save bandwidth.

[0085] It should be understood that performing face detection on images captured by the user terminal to obtain the original face image is a well-known technique in the art, and this embodiment will not provide further explanation.

[0086] It should be understood that the face difference image is obtained by subtracting the visual anonymized image from the original face image pixel by pixel. Subtracting two images pixel by pixel is a well-known technique to those skilled in the art, and this embodiment will not provide further explanation.

[0087] In one embodiment, the anonymization preprocessing includes Gaussian blur, median filtering, pixelation, and hybrid blur; wherein hybrid blur is performed by dividing an original face image into multiple image blocks and performing Gaussian blur, median filtering, and pixelation on each image block.

[0088] In this embodiment, the anonymization preprocessing is a visual pixel-level anonymization method, including but not limited to image blurring (Gaussian blur, median filtering), pixelation, and custom blending blur. The custom blending blur method divides an image into several image blocks, and each image block is processed using a different anonymization method.

[0089] It is understood that image blurring (Gaussian blur, median filtering), pixelation and other preprocessing methods are well known to those skilled in the art, and will not be described in detail in this embodiment.

[0090] This embodiment also provides simulation examples of recognition accuracy, evaluating the face recognition accuracy of the method of the present invention on five face image datasets: LFW, CALFW, CPLFW, CFP-FP, and AgeDB. The common face recognition model used is AdaFace, and the results are shown in Table 1. The method of the present invention adds four feature map fusion modules to the face recognition model AdaFace, such as... Figure 2 As shown, this is used to achieve feature fusion operation of feature map, weight parameters and visual anonymized image. Comparing the second and fourth rows of Table 1, it can be found that the recognition accuracy of anonymized image has been greatly improved. Comparing the third and fourth rows of Table 1, it can be found that the recognition accuracy of visual anonymized image in the method of the present invention is basically the same as the recognition accuracy of the user's original face image.

[0091] Table 1 shows the recognition accuracy of the method of the present invention on five datasets.

[0092]

[0093]

[0094] In summary, the embodiments of the present invention, through the above simulation examples, verify the feasibility of the face recognition method provided in this embodiment, ensure that the visually anonymized image is anonymous to the human eye, and effectively improve the identity recognition accuracy of the visually anonymized image, effectively solving the balance problem between visual privacy protection and recognition accuracy of anonymized images.

[0095] Please refer to Figure 4 , Figure 4 This diagram illustrates a structural schematic of a face recognition device for protecting visual privacy, provided by an embodiment of the present invention. This device is applied to a server, such as... Figure 4 As shown, it includes:

[0096] The receiving module 410 is used to receive a visually anonymous image sent by the user terminal and a feature map extracted by the feature extraction model deployed on the user terminal, wherein the feature map is obtained by extracting a face difference image through the feature extraction model.

[0097] The fusion recognition module 420 is used to input the visual anonymized image, feature map and weight parameters output by the feature extraction model during the training phase into a pre-trained face recognition model that is embedded with a feature fusion module for feature fusion and face recognition. When the weight parameters are correct, the face recognition model outputs a face identity feature vector and sends it to the user terminal.

[0098] The facial recognition device for protecting visual privacy in this application embodiment is similar to the one described above. Figure 1 The canvas-based data encryption transmission method shown is an invention based on the same concept. Through the above detailed description of the canvas-based data encryption transmission method, those skilled in the art can clearly understand the implementation process of the face recognition device for protecting visual privacy in this embodiment. Therefore, for the sake of brevity, it will not be described again here.

[0099] Accordingly, the face recognition device for protecting visual privacy provided in this embodiment receives visually anonymized images uploaded from user terminals on the server side, making the original face images unrecognizable by human vision, thus solving the problem of facial privacy information leakage. To improve the recognition accuracy of anonymized images, this invention receives feature maps extracted from face difference images by a feature extraction model deployed on the user terminal. Then, using a pre-trained face recognition model embedded with a feature fusion module, the feature maps and weight parameters are fused into the face recognition model during the extraction of facial identity feature vectors from visually anonymized images. The resulting facial identity feature vector can be used for accurate identity recognition. Furthermore, to ensure the security of face recognition, correct weight parameters must be used during the recognition stage of the face recognition model to output correct face recognition results; otherwise, the recognition results will be incorrect. In summary, this invention not only ensures the anonymity of visually anonymized images to the human eye but also improves the accuracy of identity recognition of visually anonymized images, thereby solving the balance problem between visual privacy protection and recognition accuracy of anonymized images.

[0100] Based on the same inventive concept as the face recognition method in the foregoing embodiments, this specification also provides a server, such as... Figure 5 As shown, please refer to Figure 5 , Figure 5 A schematic diagram of a server structure provided in an embodiment of this application is shown. The server 500 includes a processor 510, a memory 520, a communication interface 530, and at least one communication bus for connecting the processor 510, the memory 520, and the communication interface 530. The memory 520 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (PROM), or portable read-only memory (CD-ROM), and is used for related instructions and data.

[0101] The communication interface 530 is used to receive and send data. The processor 510 can be one or more CPUs; if the processor 510 is a single CPU, it can be a single-core CPU or a multi-core CPU. The processor 310 in the terminal 300 is used to read one or more programs 521 stored in the memory 520 and perform the following operations: receiving a visually anonymized image sent by the user terminal and a feature map extracted by a feature extraction model deployed on the user terminal, wherein the feature map is obtained by extracting a face difference image through the feature extraction model; inputting the visually anonymized image, the feature map, and the weight parameters output by the feature extraction model during the training phase into a pre-trained face recognition model embedded with a feature fusion module for feature fusion and face recognition; when the weight parameters are correct, the face recognition model outputs a face identity feature vector and sends it to the user terminal.

[0102] Based on the inventive concept of the face recognition method for protecting visual privacy applied to the user terminal in the foregoing embodiments, this embodiment also provides a user terminal, including a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to perform the operation as performed by the face recognition method for protecting visual privacy applied to the user terminal.

[0103] The user end includes smart terminal devices such as cameras and mobile phones, which will not be described in detail in this embodiment.

[0104] Based on the inventive concept of the face recognition method for protecting visual privacy in the foregoing embodiments, this embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods of the face recognition method for protecting visual privacy described above.

[0105] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A face recognition method for protecting visual privacy, characterized by, Applied to a server, the method comprises: Receiving a visual anonymous image sent by a user terminal and a feature map extracted by a feature extraction model deployed on the user terminal, wherein the feature map is obtained by extracting a face difference image through the feature extraction model; wherein the original face image of the user is detected and captured, and the original face image is preprocessed for anonymization of visual information to obtain the visual anonymous image; the visual anonymous image and the original face image are subtracted pixel by pixel to obtain the face difference image; the feature extraction model is trained on the server, and the trained feature extraction model is distributed to at least one user terminal to complete the deployment of the feature extraction model; the feature extraction model is modified on the basis of a MobileFaceNet network, specifically: deleting the output layer of the MobileFaceNet network, and inserting a feature map extraction module and a weight module at different positions in the MobileFaceNet network, wherein the feature map extraction module is composed of an upsampling layer and a convolution layer, and the weight module is composed of a vector matrix; the model parameters of the feature map extraction module are set according to the face recognition model; in the training stage, the feature extraction model outputs the feature map and the weight parameter; in the identification stage, the feature extraction model only outputs the feature map, and the weight parameter is used as a secret key for the face recognition model to identify; The visual anonymous image, the feature map and the weight parameter output by the feature extraction model in the training stage are input into a pre-trained face recognition model embedded with a feature fusion module for feature fusion and face recognition, and when the weight parameter is correct, the face recognition model outputs a face identity feature vector and sends it to a user end; wherein a fusion formula of the visual anonymous image, the feature map and the weight parameter is: ; wherein, represents an original face image, represents a visual anonymous image obtained by performing visual anonymization preprocessing on the original face image represents a face difference image; represents a feature map output by the feature extraction model M at the end of the i-th stage; represents a feature map output by the face recognition model F at the end of the i-th stage, represents a weight parameter, represents an activation function.​ 2. The method of claim 1, wherein, An ArcFace loss function and an L2 loss function are used to construct a total loss function, and the model parameters of the feature extraction model are updated according to the total loss function.

3. A face recognition method for protecting visual privacy, characterized by, Applied to a user terminal, a trained feature extraction model sent by a server for executing a face recognition method for protecting visual privacy according to claim 1 is received and deployed on the user terminal, the method comprising: Detecting and capturing the original face image of the user, and preprocessing the original face image for anonymization of visual information to obtain the visual anonymous image; Subtracting the visual anonymous image and the original face image pixel by pixel to obtain the face difference image; Extracting at least one group of feature maps of the face difference image according to the feature extraction model; Sending the visual anonymous image and the feature map to the server; Receiving the face identity feature vector sent by the server to complete the recognition and verification of the face of the user according to the face identity feature vector, wherein the face identity feature vector is obtained by performing feature fusion and face recognition on the visual anonymous image, the feature map, and the weight parameter output by the feature extraction model in the training stage through a pre-trained face recognition model embedded with a feature fusion module, and the face recognition model outputs when the weight parameter is correct.

4. The method of claim 3, wherein, The anonymization preprocessing includes Gaussian blur, median filtering, pixelization, and hybrid blur; wherein the hybrid blur is to divide an original face image into multiple image blocks, and perform Gaussian blur, median filtering, and pixelization processing on each image block.

5. A face recognition apparatus for protecting visual privacy, characterized by, Applied to a server, comprising: The receiving module is configured to receive a visual anonymous image sent by a user terminal and a feature map extracted by a feature extraction model deployed on the user terminal, wherein the feature map is obtained by extracting a face difference image by the feature extraction model; wherein an original face image of a user is detected and captured, and the original face image is preprocessed to be anonymous in visual information to obtain the visual anonymous image; the visual anonymous image and the original face image are subtracted pixel by pixel to obtain the face difference image; the feature extraction model is trained on the server, and the trained feature extraction model is distributed to at least one user terminal to complete the deployment of the feature extraction model; the feature extraction model is modified on the basis of a MobileFaceNet network, specifically: the output layer of the MobileFaceNet network is deleted, and a feature map extraction module and a weight module are respectively inserted at different positions in the MobileFaceNet network, wherein the feature map extraction module is composed of an up-sampling layer and a convolution layer, and the weight module is composed of a vector matrix; the model parameters of the feature map extraction module are set according to the face recognition model; in the training stage, the feature extraction model outputs the feature map and the weight parameters; in the recognition stage, the feature extraction model only outputs the feature map, and the weight parameters are used as a secret key in the recognition of the face recognition model. The fusion recognition module is configured to input the visual anonymous image, the feature map, and the weight parameter output by the feature extraction model in the training stage into a pre-trained face recognition model embedded with a feature fusion module to perform feature fusion and face recognition. When the weight parameter is correct, the face recognition model outputs a face identity feature vector and sends the face identity feature vector to the user end. A fusion formula of the visual anonymous image, the feature map, and the weight parameter is as follows: ; wherein, represents an original face image, represents a visual anonymous image obtained by performing visual anonymization preprocessing on the original face image, represents a face difference image, represents a feature map output by the feature extraction model M at the end of the i-th stage, represents a feature map output by the face recognition model F at the end of the i-th stage, represents a weight parameter, represents an activation function. represents an activation function.

6. A server, characterized by The server comprises a processor and a memory, and the memory stores at least one instruction, which is loaded and executed by the processor to implement the operations performed by the face recognition method for protecting visual privacy according to any one of claims 1 to 2.

7. A user terminal, characterized in that The user terminal comprises a processor and a memory, and the memory stores at least one instruction, which is loaded and executed by the processor to implement the operations performed by the face recognition method for protecting visual privacy according to any one of claims 3 to 4.

Images