A face verification system based on lensless camera

Abstract

The present invention relates to a face verification system based on a lensless camera. The system comprises a lensless mask, a camera sensor, and a controller. The lensless mask is used for lensless imaging of a face, and its parameters are optimizable. The camera sensor is used to capture a face image formed after passing through the lensless mask. The controller is used to directly perform face verification based on the face image captured by the camera sensor. The training process of the controller includes: establishing an imaging model for the lensless mask; inputting the face image into the imaging model to obtain a lensless imaging result; then inputting the result into a face verification model to be trained to obtain a face identity prediction result; calculating a loss function based on the face identity prediction result, thereby jointly optimizing and training the imaging model and the face verification model; and the face verification system optimizes the parameters of the lensless mask based on the trained imaging model. Compared with the existing technology, the present invention enhances the security of face data and improves prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of face recognition, and in particular to a face verification system based on a lensless camera. Background Art

[0002] Facial recognition technology, a key branch of computer vision, is widely used in fields such as access control, retail, and smartphone security. These applications often rely on large-scale facial image datasets for facial recognition and verification. Despite significant advances in facial recognition technology, the potential for facial image leakage during the collection and storage of facial images has become an increasingly prominent issue.

[0003] To protect user privacy in facial recognition applications, researchers have proposed using lensless cameras to capture facial images. Lensless cameras encrypt the original scene information at the hardware level during capture, resulting in optically encrypted images that effectively protect user privacy.

[0004] Existing lensless face recognition systems typically use a "reconstruction-recognition" process. The encoded image captured by a lensless camera is first reconstructed into the original facial image, and then traditional RGB-based face recognition techniques are applied. However, this method is vulnerable to software attackers during reconstruction, resulting in the leakage of the reconstructed facial image. In addition, the patterns of lensless optical encoders designed for imaging are not optimized for face recognition tasks, which in turn reduces face recognition accuracy. Furthermore, the additional reconstruction step increases computational cost, which poses a challenge for its application in scenarios with limited computing resources. Summary of the Invention

[0005] The purpose of the present invention is to provide a face verification system based on a lensless camera in order to overcome the defect that the above-mentioned prior art adopts a "reconstruction-recognition" process, which is vulnerable to attacks by software attackers during reconstruction, thereby leading to the leakage of the reconstructed facial image.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A face verification system based on a lensless camera includes a lensless mask, a camera sensor, and a controller. The lensless mask is used to perform lensless imaging of a face, the camera sensor is used to collect a face image formed after passing through the lensless mask, and the controller is used to directly perform face verification based on the face image collected by the camera sensor.

[0008] Furthermore, the parameters of the lensless mask can be optimized, and the processing of the controller further includes the following steps:

[0009] Establish an imaging model for the lensless mask and construct a face verification model to be trained;

[0010] Inputting a face image into the imaging model to obtain a lens-free imaging result, which is then input into a face verification model to be trained to obtain a face identity prediction result, and calculating a loss function based on the face identity prediction result, thereby jointly optimizing and training the imaging model and the face verification model;

[0011] Use the trained face verification model for face verification;

[0012] The face verification system optimizes parameters of the lensless mask according to the trained imaging model.

[0013] Furthermore, the construction process of the imaging model includes: obtaining an impulse response function corresponding to the lens-free mask based on the parameters of the lens-free mask, using the impulse response function as a convolution kernel to construct the imaging model, and the imaging model performs a convolution operation on the input face image based on the convolution kernel to obtain a lens-free imaging result.

[0014] Furthermore, the processing of the controller further includes:

[0015] The lensless imaging result output by the imaging model is subjected to face center alignment and then input into the face verification model. The face center alignment includes: obtaining the face center position according to the lensless imaging result output by the imaging model, performing center cropping on the lensless imaging result based on the face center position, and obtaining the lensless imaging result after alignment with the face center.

[0016] Furthermore, in the joint optimization training, the input facial image is first transformed and augmented and then input into the imaging model for training. Then, during the training process, the degree of the augmentation variable is gradually increased and the obtained facial image is input into the imaging model for training.

[0017] Furthermore, the transformation and augmentation method includes rotation, scaling and background setting.

[0018] Furthermore, the joint optimization training process also includes: inputting the facial image input into the imaging model into a pre-trained RGB face verification model at the same time to obtain a soft label corresponding to the facial image; respectively comparing the facial identity prediction result obtained by the face verification model with the soft label corresponding to the facial image and the real identity label corresponding to the facial image, calculating two training losses, and performing joint optimization of the imaging model and the face verification model based on the two calculated training losses.

[0019] Furthermore, the RGB face verification model is a face verification model based on face images in RGB mode.

[0020] Furthermore, the face verification model is a deep learning model.

[0021] Furthermore, the lensless mask is an amplitude-based optical mask or a phase-based optical mask.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] (1) The present invention proposes to directly perform face identity authentication using the results of lens-free imaging, rather than performing face authentication by first reconstructing and then verifying the results of lens-free imaging. This avoids the unnecessary consumption of computing resources and privacy leakage problems that may be caused by the reconstruction step. In the absence of a decoding mode, the image content is unrecognizable to a third party, which significantly enhances the security of facial data and the privacy protection effect of the lens-free face verification system.

[0024] (2) The present invention proposes to perform end-to-end optimization of the lensless face verification system and jointly optimize the lensless optical mask (hardware part) and the lensless face verification model (software part), rather than just optimizing the lensless face verification model. This can obtain an optical mask that is more conducive to face recognition tasks, thereby improving the accuracy of the lensless face verification system.

[0025] (3) The present invention proposes an overall end-to-end optimization method for a lens-free face verification system, which includes three important key technical points, namely, a lens-free face center alignment method, a data augmentation strategy based on curriculum learning, and a training loss based on cross-modal distillation. It has a certain robustness to faces at different positions and angles in the scene, and distills the knowledge in an RGB modality face verification model into a lens-free modality face verification model, thereby improving training efficiency and prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of a verification process of a lensless camera-based face verification system provided in an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of a face center alignment process based on lensless imaging, provided in an embodiment of the present invention. The image in scene space undergoes lensless imaging simulation, i.e., convolution is performed on the impulse response function corresponding to the lensless optical mask to obtain a lensless imaging result in sensor space. This lensless imaging result is then cropped for alignment. The upper and lower rows in the figure illustrate the process of aligning faces before and after translation in scene space.

[0028] Figure 3 This is a schematic diagram of a lensless camera prototype provided in an embodiment of the present invention. The lensless camera consists of a lensless mask and a camera sensor. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0030] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0031] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0032] Existing lensless face recognition systems typically use a "reconstruction-recognition" process. The encoded image captured by a lensless camera is first reconstructed into the original facial image, and then traditional RGB-based face recognition techniques are applied. However, this method is vulnerable to software attackers during reconstruction, resulting in the leakage of the reconstructed facial image. In addition, the patterns of lensless optical encoders designed for imaging are not optimized for face recognition tasks, which in turn reduces face recognition accuracy. Furthermore, the additional reconstruction step increases computational cost, which poses a challenge for its application in scenarios with limited computing resources.

[0033] This invention aims to identify faces directly from optically encrypted images captured by a lensless camera, thus avoiding the unnecessary computational resource consumption and privacy concerns associated with the reconstruction step. Furthermore, to improve the performance of face recognition systems, this invention employs an end-to-end optimization approach for lensless optical encoding reticles and deep learning-based lensless face recognition models for face recognition tasks.

[0034] Example 1

[0035] like Figure 1 As shown, this embodiment provides a face verification system based on a lensless camera, including a lensless mask, a camera sensor and a controller. The lensless mask is used to perform lensless imaging of the face, the camera sensor is used to collect the face image formed after passing through the lensless mask, and the controller is used to directly perform face verification based on the face image collected by the camera sensor.

[0036] That is, the goal of this embodiment is to establish an end-to-end optimized lensless camera-based face verification system. This system optimizes the optical lensless mask and the electrical face verification model to achieve the best face verification effect. To improve efficiency and protect privacy, the present invention directly performs face verification based on the encoded image obtained by lensless camera imaging. The face verification process of this system is as follows: Figure 1 shown.

[0037] Preferably, to achieve optimization of the end-to-end lensless face verification system, the parameters of the lensless mask can be optimized, and the processing of the controller further includes the following steps:

[0038] Establish an imaging model for the lensless mask and construct a face verification model to be trained;

[0039] The face image is input into the imaging model to obtain the lens-free imaging result, which is then input into the face verification model to be trained to obtain the face identity prediction result. The loss function is calculated based on the face identity prediction result, thereby jointly optimizing the imaging model and the face verification model.

[0040] Use the trained face verification model for face verification;

[0041] The face verification system optimizes the parameters of the lensless mask based on the trained imaging model.

[0042] The imaging model construction process includes: obtaining the impulse response function corresponding to the lensless mask based on the parameters of the lensless mask, using the impulse response function as the convolution kernel to construct the imaging model, and the imaging model performs a convolution operation on the input face image based on the convolution kernel to obtain the lensless imaging result.

[0043] In other words, in order to utilize a large amount of RGB face recognition data sets to optimize an end-to-end lens-free face verification system, the present invention first models the imaging process of a lens-free camera, and uses the modeling to simulate the imaging results of a lens-free camera with a parameter-optimizable optical mask on the face, and then inputs the simulated imaging results into the face verification model to be trained to obtain the identity prediction result of the face image by the face verification model, and finally inputs the identity prediction result and the true identity of the face image into the loss function of the face verification model, and uses the loss function to train the face verification model and the optimizable lens-free optical mask end-to-end.

[0044] To enable end-to-end optimization of the system, the present invention requires that the modeling of the lensless imaging process be differentiable. Specifically, the present invention utilizes a convolution model to model the lensless imaging process. The present invention first obtains the impulse response function corresponding to the optical mask based on the parameters of the mask. This impulse response function is then used as a convolution kernel, which is applied to the RGB facial image. The convolution result is the final simulated imaging result of the lensless camera on the face.

[0045] Preferably, in order to train a robust face verification system that can handle translation, rotation and background changes, and to achieve better optimization results for the face verification system, the present invention proposes a new training process comprising three important components: 1) Before the face verification model, the present invention proposes a new face center alignment algorithm based on a lens-free imaging model to handle the problem of facial translation; 2) In order to improve the robustness of the face verification model under rotation and background changes, the present invention proposes a new data augmentation strategy based on curriculum learning. 3) In order to improve the accuracy of the face verification system, the present invention proposes a new training loss function based on cross-modal distillation. Next, the present invention elaborates on the specific technical implementation schemes of these three parts.

[0046] 1. New face center alignment algorithm based on lensless imaging model

[0047] The lensless imaging result output by the imaging model is subjected to face center alignment and then input into the face verification model. The face center alignment includes: obtaining the face center position according to the lensless imaging result output by the imaging model, center cropping the lensless imaging result based on the face center position, and obtaining the lensless imaging result after alignment with the face center.

[0048] Specifically, two spaces are defined (such as Figure 2 (as shown in the figure): “scene space” and “sensor space”. The natural scene including the face before being photographed by the lensless camera is located in the “scene space”, while the image captured by the sensor after the light passes through the lensless optical mask is located in the “sensor space”. The process of obtaining the image in the “sensor space” from the “scene space” through the lensless optical mask is simulated by the convolution operation, and the convolution kernel of the convolution operation is the impulse response function corresponding to the lensless optical mask (as shown in the figure). Figure 2 shown).

[0049] Face center alignment is applied in sensor space. This alignment transforms lensless camera imaging results of faces to a canonical position, reducing the training and testing gap and improving the accuracy of the face verification system for non-centered faces. Specifically, the present invention estimates the face center position based on the imaging results in sensor space, and then performs center cropping on the face center to obtain the lensless imaging result after alignment.

[0050] 2. Data augmentation strategy based on curriculum learning

[0051] In the joint optimization training, the input facial image is first transformed and augmented and then input into the imaging model for training. Then, during the training process, the degree of the augmentation variable is gradually increased, and the obtained facial image is input into the imaging model for training. The transformation and augmentation methods include rotation, scaling and background setting.

[0052] Specifically, in order to enable the face verification system based on a lensless camera to be robust to various transformations (rotation, scaling and background), the present invention proposes to use face images augmented with various transformations for training during end-to-end training, and control the degree of enhancement variables such as rotation angle and background size during the training process, and gradually increase these augmentations, so that the model can preferentially learn the ability of face verification through simple samples at the beginning of training, and gradually become robust to different changes through more challenging samples.

[0053] 3. New training loss function based on cross-modal distillation

[0054] The face image input to the imaging model is simultaneously input into a pre-trained RGB face verification model to obtain the soft label corresponding to the face image; the face identity prediction result obtained by the face verification model is respectively compared with the soft label corresponding to the face image and the real identity label corresponding to the face image, and two training losses are calculated. The imaging model and the face verification model are jointly optimized based on the two calculated training losses.

[0055] The RGB face verification model is a face verification model based on face images in RGB mode.

[0056] The calculation of this loss function distills the knowledge from an RGB face verification model into a lensless face verification model. Specifically, the present invention uses a trained RGB face verification model to obtain "soft labels" for facial images. It then calculates the loss between these "soft labels" and the identity predictions output by the lensless face verification model, and uses this loss function to optimize the lensless face verification system.

[0057] The above face verification model can be any deep learning model, such as Resnet, VIT.

[0058] The lensless mask may be any optical mask whose parameters can be optimized, including an amplitude-based optical mask or a phase-based optical mask.

[0059] The overall training process of this scheme includes: a face image containing background and rotation transformations is subjected to a lens-free imaging process (convolution is performed on the impulse response function of the lens-free optical mask, and the lens-free optical mask of this process is optimizable) to obtain a lens-free imaging result, and the face center is aligned on the lens-free imaging result. The aligned lens-free image is input into the lens-free face verification model to be trained to obtain a face identity prediction result. The result and the "soft label" output by the RGB modality face verification model and the true identity label of the face are used to obtain two training losses respectively, and the training loss is used to jointly optimize the lens-free face verification model and the lens-free optical mask.

[0060] The following example builds a lens-free face verification prototype system to demonstrate the feasibility of the method proposed in this invention. Figure 3 As shown in Figure 1, the prototype system includes a screen that can display facial scenes, an optimized lensless optical mask, and a camera sensor. By displaying different facial scenes on the screen, the system achieved a face verification accuracy of 91.53% on the commonly used face verification performance evaluation benchmark LFW.

[0061] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A face verification system based on a lensless camera, characterized in that: It includes a lensless mask, a camera sensor, and a controller. The lensless mask is used to perform lensless imaging of a human face. The camera sensor is used to collect a facial image formed after passing through the lensless mask. The controller is used to directly perform facial verification based on the facial image collected by the camera sensor. The parameters of the lensless mask can be optimized, and the processing of the controller further includes the following steps: Establish an imaging model for the lensless mask and construct a face verification model to be trained; Inputting a face image into the imaging model to obtain a lens-free imaging result, which is then input into a face verification model to be trained to obtain a face identity prediction result, and calculating a loss function based on the face identity prediction result, thereby jointly optimizing and training the imaging model and the face verification model; Use the trained face verification model for face verification; The face verification system optimizes parameters of the lensless mask according to the trained imaging model; The imaging model construction process includes: obtaining an impulse response function corresponding to the lensless mask according to the parameters of the lensless mask, using the impulse response function as a convolution kernel to construct the imaging model, and performing a convolution operation on the input face image based on the convolution kernel to obtain a lensless imaging result; The processing of the controller also includes: Performing face center alignment on the lens-free imaging result output by the imaging model, and then inputting the result into the face verification model, wherein the face center alignment includes: obtaining a face center position according to the lens-free imaging result output by the imaging model, and performing center cropping on the lens-free imaging result based on the face center position to obtain a lens-free imaging result after alignment with the face center; The joint optimization training process also includes: inputting the facial image input to the imaging model into a pre-trained RGB face verification model to obtain a soft label corresponding to the facial image; respectively comparing the facial identity prediction result obtained by the face verification model with the soft label corresponding to the facial image and the real identity label corresponding to the facial image, calculating two training losses, and jointly optimizing the imaging model and the face verification model based on the two calculated training losses.

2. A face verification system based on a lensless camera according to claim 1, characterized in that: In the joint optimization training, the input facial image is first transformed and augmented and then input into the imaging model for training. Then, during the training process, the degree of the augmentation variable is gradually increased and the obtained facial image is input into the imaging model for training.

3. The face verification system based on a lensless camera according to claim 2, characterized in that: The transformation augmentation method includes rotation, scaling and background setting.

4. The face verification system based on a lensless camera according to claim 1, characterized in that: The RGB face verification model is a face verification model based on face images in RGB mode.

5. The face verification system based on a lensless camera according to claim 1, characterized in that: The face verification model is a deep learning model.

6. The face verification system based on a lensless camera according to claim 1, characterized in that: The lensless mask is an amplitude-based optical mask or a phase-based optical mask.

Images