Continuous learning method for facial comparison biometric neural networks

Abstract

Continuous Learning Method for a Facial Comparison Biometrics Neural Network A continuous learning method for a facial comparison biometrics neural network includes the following operations: a) receiving a pre-trained facial comparison biometrics neural network, b) splitting the neural network from operation a) into a plurality of consecutive layers starting from the first layer defining a characterization block, and one or more fully connected layers defining an output block, c) splitting the characterization block into an extraction block comprising a chosen number of consecutive layers starting from the first layer of the characterization block, and a fitting block comprising the remaining layers of the characterization block, d) receiving two distinct sets of training data formatted as the inputs to the neural network from operation a), the first set comprising non-confidential data,and the second set comprising confidential data, e) provide the second set to the extraction block (230) and extract a third set from it, and f) train the neural network for operation a) by freezing the extraction block (230) using the first dataset with the extraction block (230) frozen, and by using the third set by providing it directly to the adaptation block. Fig. 1,

Description

Title of the invention: Continuous learning method for biometric neural network by facial comparison

[0001] The invention relates to the field of identification and authentication, and in particular the field of biometrics by facial comparison.

[0002] The principle of these applications is to record or process in real time a selfie video of a user, in front of his computer or his mobile phone, who is carrying out an authentication or identity verification process.

[0003] This video is analyzed by algorithms and / or human operators whose role is to authenticate the user, or to validate whether the user corresponds to an identity document provided in a remote identification process.

[0004] The last decade has seen significant evolution in the field of deep learning and an increase in the availability of benchmark datasets allowing the community to train more complex systems and achieve state-of-the-art performance.

[0005] The field of computer vision applied to facial (or face) images has benefited from this rapid development and has facilitated the adoption of deep learning techniques in facial biometrics (e.g., facial recognition, live detection).

[0006] These modern techniques excel in learning powerful models from fixed datasets and stationary environments, and regularly surpass human capabilities. However, these models fail to emulate the human learning process, which is efficient, robust, and capable of learning incrementally from sequential experiences in a non-stationary world.

[0007] To adapt neural networks to a constantly changing environment, two options are possible:

[0008] - Retraining: updating the datasets used during training by adding the new data and restarting training from scratch on the updated datasets.

[0009] - Continuous learning: updating the data sets used during training by adding new data and fine-tuning existing neural networks on the updated datasets.

[0010] These techniques, if used correctly, tend to work and allow for lifelong learning. However, they require access to data sets of the same nature as those on which existing neural networks have been trained, which is impossible in practice.

[0011] The invention improves the situation. To this end, it proposes a continuous learning method for a biometric neural network by facial comparison comprising the following operations: a) receiving an already trained facial comparison biometrics neural network, b) separating the neural network of operation a) into a plurality of consecutive layers from the first layer defining a characterization block, and one or more fully connected layers defining an output block, c) separating the characterization block into an extraction block comprising a selected number of consecutive layers from the first layer of the characterization block, and an adaptation block comprising the other layers of the characterization block, d) receiving two separate sets of training data formatted as the inputs to the neural network of operation a), the first set comprising non-confidential data, and the second set comprising confidential data, e) providing the second set to the extraction block and drawing a third set from it, and f) training the neural network of operation a) by freezing the extraction block using the first dataset with the frozen extraction block, and using the third set by providing it directly to the adaptation block.

[0012] This method is particularly advantageous because it allows for continuous training of an already trained neural network. In addition, this device makes it possible to make part of the data confidential, by keeping only their characteristics as seen by the extraction block.

[0013] According to various embodiments, the invention may have one or more of the following characteristics: - operation c) includes choosing the size of the extraction block based on the weighted sum of an AUC distinction score, a complexity score and an optional disk space score, - the distinction score is an area under the curve score, and the complexity score is derived from the number of parameters in the extraction block, - the weighted sum is calculated as (4*AUC+2*Complexity Score) / 6, - the weighted sum is calculated as (4*AUC+2*Complexity Score+0.5*Disk Space Score) / 6.5, - operation c) includes choosing the size of the extraction block based on the weighted sum of a normalization score, a distinction score, a complexity score and an optional disk space score, - the normalization score is the application surface of the normalization layers, and the distinction score is a BPCER type score, - the weighted sum is calculated as (Normalization Score+4*Distinction Score+2*Complexity Score) / ?, and - the weighted sum is calculated as (Normalization Score+4*Distinction Score+2*Complexity Score+Disk Space Score) / 8.

[0014] The invention also relates to a computer program comprising instructions for executing the method according to the invention, a data storage medium on which such a computer program is recorded and a computer system comprising a processor coupled to a memory, the memory having recorded such a computer program.

[0015] Other characteristics and advantages of the invention will appear more clearly on reading the following description, taken from examples given for illustrative and non-limiting purposes, taken from the drawings in which: - [Fig.l] represents a schematic diagram of a method according to the invention implemented on an already trained facial comparison biometric neural network, and - [Fig.2] represents an example of a neural network which is the subject of the method of [Fig.l],

[0016] The drawings and the description below contain, for the most part, elements of a certain character. They may therefore not only serve to better understand the present invention, but also contribute to its definition, where appropriate.

[0017] [Fig.l] represents a schematic diagram of a method according to the invention implemented on an already trained facial comparison biometric neural network.

[0018] In a first step 100, the already trained facial comparison biometric neural network is received. In a known manner, this type of neural network is a deep learning neural network, for example from the CNN family (convolutional neural network), RNN family (recurrent neural network), RCNN family (recurrent convolutional neural network), etc.

[0019] A common characteristic of all these architectures is the creation of the network in two portions: - an input vector characterization block, which extracts the features via a succession of layers that can be of various types such as convolutional, pooling (or "pooling rent" in English), correction (for example ReLU, sigmoid, etc.), and - an output block comprising one or more fully connected layers, i.e. whose neurons have connections to all the outputs of the previous layer.

[0020] Generally speaking, the feature portion is considered to extract all the features from the input vector according to the network, while the output layer performs the high-level reasoning of the neural network. Alternatively, these portions are also known as "backbone" for the characterization portion, and "head" for the output portion.

[0021] This distinction is important because the invention applies when working on the backbone. Indeed, when a neural network has been trained, and we want to update its learning, several problems arise. Thus, the heterogeneity of the new data penalizes the use of continuous learning compared to retraining from scratch, but in addition there is the problem of the right to use new training data.

[0022] This last topic is particularly important in the context of facial comparison biometric neural networks. Indeed, all executions of the already trained neural network offer an almost inexhaustible source of new data. But the input vectors of these, images or biometric video streams are by nature very sensitive and cannot be used directly. And when it is not the laws protecting privacy that prohibit the use of this data, it is the managers of these neural networks who do not want to take the risk of being accused of not respecting the privacy of their users.

[0023] In order to take into account the need to carry out continuous learning while preserving the confidentiality and privacy of users, the Applicant had the idea of ​​decomposing the already trained backbone of its facial comparison biometric neural networks into two parts, as shown in [Fig. 2]. As can be seen in [Fig. 2], a facial comparison biometric neural network 200 comprises a backbone 210 and a head 220. The backbone or characterization block 210 is itself divided into an extraction block 230 and an adaptation block 240. By definition, the extraction block 230 comprises the first layers of the characterization block 210, starting from the input layer, and the adaptation block 240 comprises the rest of the layers of the characterization block 210.

[0024] In an operation 110, the characterization block 210 is separated into an extraction block 230 and an adaptation block 240. The extraction block 230 is then frozen in an operation 120, which means that the continuous learning of [Fig.l] will not change the weightings and parameters of the layers that constitute the characterization block 230, which will therefore remain identical to what it was in the facial comparison biometric neural network 200 received as input.

[0025] The data to be used to carry out continuous learning but whose confidentiality is desired to be preserved are then processed by the extraction block 230 in an operation 120. This results in a plurality of confidentialized vectors, which are formatted to be able to be provided as input to the adaptation block 240, but which are no longer comparable to the input vectors from which they were drawn. Indeed, it is known that the first layers of deep neural networks very strongly affect the input vectors, and that the outputs of these do not allow a link to be made with the input vector from which these outputs are drawn.

[0026] Therefore, by appropriately sizing the characterization block 230, it becomes possible to perform continuous training of the facial comparison biometric neural network 200 from this data which has been confidentialized.

[0027] Thus, in an operation 130, the facial comparison biometric neural network 200 is trained again by leaving the extraction block 230 frozen, and by training the adaptation block 240 and the head 220. This is achieved by providing the vectors from the operation 120 directly to the adaptation block 240, and by providing input vectors not requiring confidentiality as input to the characterization block 230.

[0028] The larger the extraction block 230, i.e. the greater the number of layers it contains, the greater the confidentiality of the input vectors to be made confidential, but to the detriment of the quality of the continuous learning carried out, since the extraction block 230 is fixed. Conversely, it is not desirable for the extraction block 230 to be too small in order to preserve confidentiality.

[0029] The Applicant has therefore carried out research in order to characterize the parameters which can enable it to best size the extraction block 230. Thus, in a first implementation of the method according to the invention, the Applicant has focused on the following parameters: - a distinction score, which characterizes the model. In the example described here, this is the Area Under the Curve (AUC) score. AUC is a metric used to quantify the performance of a binary classification model. It is calculated as the ROC (Receiver Operating Characteristic) curve, which is a graphical representation of a model's ability to distinguish between positive and negative data, and to distinguish between positive and negative classes. A higher AUC score indicates better discrimination, with 1.0 representing perfect classification and 0.5 indicating random guessing. - a complexity score, which can be derived from the number of parameters in the extraction block 230. In the example described here, this is the base 10 logarithm of the number of parameters in the extraction block 230. Alternatively, it could be the logarithm in another base, or another variant with exponential decay. - an optional disk space score. In the example described here, this is the difference between the disk space originally occupied by the images to be confidentialized and the disk space occupied by the vectors representing this data after it has been confidentialized.

[0030] The Applicant's various research studies have led it to consider that the composite score formula for determining the most favorable value of the number of layers for the extraction block is obtained by the formula

[0031] Composite Score = (4*AUC+2*Complexity Score+0.5*Disk Space Score) / 6.5

[0032] As mentioned, the disk space score can be ignored, which brings the normalization factor down to 6 instead of 6.5.

[0033] To demonstrate the effectiveness of this continuous learning, the Applicant tested it on two architectures.

[0034] The first architecture uses a "SwinV2" type network, which is an improvement of the "Visual Transformer" neural network, in "Small" version, with characteristics of a window size equal to 7 and a resolution of 224, and an average normalization with 25 normalization layers. The article by Liu, Ze, et al. "Swin transformer v2: Scaling up capacity and resolution", Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, https: / / openaccess.thecvf.com / content / CVPR2022 / papers / Liu_Swin_Transformer_V2_Scaling_Up_Capacity_and_Resolution_CVPR_2022_paper.pdf describes this type of network.

[0035] The second architecture uses a network of type "EfficientNetV2", which is a convolutional neural network, in "Medium" version, with high normalization with 110 normalization layers. The article by Tan, Mingxing, and Quoc Le. "Efficientnetv2: Smaller models and faster training", International conference on machine learning. PMLR, 2021, http: / / proceedings.mlr.press / vl39 / tan21a / tan21a.pdf describes this type of network.

[0036] In order to simulate a continuous learning environment, the training dataset was divided into two: 90% for basic training of the architectures, and 10% to train vectors to be confidentialized and perform continuous learning. After initial training was performed, continuous learning was performed with the following parameters: at least 50% of trainable layers in total, the trainable layers following the extraction block 230 up to 50% being modified only on their normalization, use of the AdamW optimizer with a learning rate of 105 and a block size of 32, without any operation of dropout (operation to prevent overfitting). Loshchilov, Ilya, and Hutter, Frank, "Fixing weight decay regularization in Adam," 2018, https: / / openreview.net / pdf?id=rk6qdGgCZ, describes this optimizer.

[0037] The following results were obtained: Architecture Configuration (ES = Relative Extract Block Size) A UC Score Complexity Score Disk Space Score Composite Score EfficientNetV 2 ES = 20% No Dropout 100 86 0 88 EfficientNetV 2 ES = 30% No Dropout 87 100 100 92 Swinv2 ES = 20% No Dropout 95 86 0 85 Swinv2 ES = 30% No Dropout 100 100 100 100

[0038] The scores are of course normalized and represent the normalization out of 100 of each score relative to the best of all scores.

[0039] This table shows that an extraction block size equal to 30% is the most favorable for both types of architecture.

[0040] The table below compares the performance of the network with continuous learning versus the same network trained from scratch with all the data (including confidential data which makes this method unusable), and versus the same network without retraining. Architecture Configuration Score AUC EfficientNetV2 Baseline - no retraining 99% EfficientNetV2 Continuous Learning 99.41% EfficientNetV2 ES = 30% No Dropout 99.27% ​​Visual Transformer Baseline - no retraining 99.27% Visual Transformer Continuous Learning Classic 99.77% Visual Transformer ES = 30% No Dropout 99.75%

[0041] It appears first of all that continuous learning (classical or according to the invention) always improves performance compared to the baseline. By comparing the two continuous learning methods, we note the drop in performance of continuous learning according to the invention compared to classic retraining. This was nevertheless predictable due to the impossibility of using a data augmentation technique within the framework of the invention, since the original vectors are no longer accessible once confidentialized.

[0042] This table proves that the invention makes it possible to exploit new data even if access to the original vectors is not possible in order to maintain peak performance while preserving the privacy of users.

[0043] In a second implementation of the method according to the invention, the Applicant focused on other parameters: - a normalization score. In the example described here, this is the area of ​​application of the normalization layers, for example the number of normalization layers corresponding to the extraction block 230. - a distinction score, which characterizes the model. In the example described here, this is the BPCER score (for "Bonafide presentation classification error rate" in English). The higher the BPCER score, the higher the number of false negatives. This score is supplemented by the APCER score (for "Attack presentation classification error rate" in English), which is representative of the number of false positives, i.e. vectors accepted by the neural network when they should not have been. In the example described here, the value 100-BPCER is used. - a complexity score, which can be derived from the number of parameters in the extraction block 230. In the example described here, this is the logarithm in base 10 of the number of parameters in the extraction block 230. Alternatively, this could be the logarithm in another base, or another variant with exponential decay. - an optional disk space score. In the example described here, this is the difference between the disk space originally occupied by the images to be confidentialized and the disk space occupied by the vectors representing this data after it has been confidentialized.

[0044] The Applicant's various research studies have led it to consider that the composite score formula for determining the most favorable value of the number of layers for the extraction block is obtained by the formula

[0045] Composite Score = (Normalization Score+4*Distinction Score+2*Complexity Score+Disk Space Score) / 8

[0046] As mentioned, the disk space score can be ignored, which brings the normalization factor down to 7 instead of 8.

[0047] To demonstrate the effectiveness of this continuous learning, the Applicant tested it on two architectures.

[0048] The first architecture uses a "Visual Transformer" type network, in "Base" version, with characteristics of a pattern size equal to 16 and a resolution of 224, and an average normalization with 25 normalization layers.

[0049] The second architecture uses a network of type "Efficient NelV2'\ which is a convolutional neural network, in "Medium" version, with high normalization with 110 normalization layers.

[0050] In order to simulate a continuous learning environment, the training dataset was split in two: 95% for basic training of the architectures, and 5% to form vectors to be confidentialized and perform continuous learning. After initial training was completed, continuous learning was performed with the following parameters: at least 50% of trainable layers in total, the trainable layers following the extraction block 230 up to 50% being modified only on their normalization, use of the SGD optimizer (see for example https: / / en.wikipedia.org / wiki / Stochastic_gradient_descent) with a learning rate of 10 4 and a block size of 32, without dropout operation (operation to prevent overfitting).

[0051] The following results were obtained: Architecture Configuration Normalization score Distinction score Complexity score Disk space score Composite score EfficientNet V2 ES = 20% No Dropout 100 99 85 100 96 EfficientNet V2 ES = 30% No Dropout 87 100 92 50 90 EfficientNet V2 ES = 40% No Drop out 72 90 95 62 85 EfficientNet V2 ES = 50% No Drop out 57 96 100 62 88 Visual Trans former ES = 20% No Drop out 100 100 77 100 94 Visual Trans former ES = 30% No Drop out 84 97 94 100 95 Visual Trans former ES = 40% No Drop out 68 96 98 100 93 Visual Trans former ES = 50% No Drop out 52 97 100 100 92

[0052] Here again, the scores are of course normalized and represent the normalization out of 100 of each score relative to the best of all scores.

[0053] This table shows that an extraction block size equal to 20% is the most favorable for the Visual Transformer network and equal to 30% is the most favorable for the EfficientNetV2 network, which corresponds to the formula explained above.

Claims

Claims

1. A continuous learning method for a facial comparison biometrics neural network comprising the following operations: a) receiving an already trained facial comparison biometrics neural network, b) separating the neural network of operation a) into a plurality of consecutive layers from the first layer defining a characterization block, and one or more fully connected layers defining an output block, c) separating the characterization block into an extraction block comprising a selected number of consecutive layers from the first layer of the characterization block, and an adaptation block comprising the other layers of the characterization block, d) receiving two separate sets of training data formatted as the inputs to the neural network of operation a), the first set comprising non-confidential data, and the second set comprising confidential data,e) providing the second set to the extraction block (230) and drawing a third set therefrom, and f) training the neural network of operation a) by freezing the extraction block (230) using the first data set with the extraction block (230) frozen, and using the third set by providing it directly to the adaptation block.,

2. The method of claim 1, wherein step c) comprises choosing the size of the extraction block based on the weighted sum of an AUC distinction score, a complexity score, and an optional disk space score.

3. The method of claim 2, wherein the distinctiveness score is an area under the curve score, and the complexity score is derived from the number of parameters in the extraction block.

4. The method of claim 2 or 3, wherein the weighted sum is calculated as (4*AUC+2*Complexity Score) / 6.

5. The method of claim 2 or 3, wherein the weighted sum is calculated as (4*AUC+2*Complexity Score +0.5*Disk Space Score) / 6.

5.

6. The method of claim 1, wherein step c) comprises choosing the size of the extraction block based on the weighted sum of a normalization score, a distinctiveness score, a complexity score, and an optional disk space score.

7. The method of claim 6, wherein the normalization score is the application area of ​​the normalization layers, and the distinction score is a BPCER ("Bonafide presentation classification error rate") type score.

8. The method of claim 6 or 7, wherein the weighted sum is calculated as (Normalization Score+4*Distinction Score+2*Complexity Score) / 7.

9. The method of claim 6 or 7, wherein the weighted sum is calculated as (Normalization Score+4*Distinction Score+2*Complexity Score+Disk Space Score) / 8.

10. A computer-implemented computer program comprising instructions for executing the method according to one of the preceding claims.

11. Data storage medium on which the computer program according to claim 10 is recorded.