method for generating an augmented image and associated device

Abstract

A method comprising: for each image in a plurality of images showing an individual, applying (104) a processing technique to the image to produce a biometric template for that individual; calculating (106) an average template constituting the average of the biometric templates produced; modifying (110) a selected image (I0) from the plurality of images into a modified image (I1), the modification (110) being adapted so that an error between the average template (TM) and a biometric template produced by applying the processing technique to the modified image is less than an error between the average template (TM) and the biometric template produced by applying the processing technique to the selected image (I0). Figure for the abstract: Fig. 2

Description

Title of the invention: Method for generating an augmented image and associated device. FIELD OF THE INVENTION

[0001] The present invention relates to the field of biometrics.

[0002] The present invention relates particularly to a method comprising generating an augmented image from a plurality of images of an individual. PRIORITY OF THE TECHNOLOGY

[0003] A known biometric authentication system comprises two devices: a first device comprising an image sensor for acquiring images, and a second device producing an authentication result, the first device and the second device communicating via a communication network.

[0004] A first method implemented in such a system is as follows: the image sensor acquires an image showing the individual. The image is transmitted via the communication network to the second device. The second device produces a biometric template from the transmitted image. This biometric template can then be compared with reference templates stored in a database. If there is a match between the calculated biometric template and one of the reference templates, the individual is considered an authorized user. However, this first method can lead to an erroneous result if the acquired image is noisy, in particular if the image is blurry or does not show a distinguishing part of the user's body with sufficient clarity.

[0005] To overcome this problem, a second method has been proposed in which the image sensor acquires not a single image but several, and the processing implemented during the first method is applied to each image. This second method, however, has several drawbacks: firstly, it consumes more bandwidth than the first method (since several images are transmitted over the communication network instead of just one); secondly, it requires a greater computational load than the first method. Description of the invention

[0006] An object of the invention is to generate data relating to an individual on the basis of which biometric authentication can be reliably carried out and consuming little bandwidth when transmitted in a communication network.

[0007] To this end, a computer-implemented method for generating an augmented image is proposed according to a first aspect, the method comprising: - For each image that is part of a plurality of images showing an individual, the image is processed to produce a biometric template relating to the individual. - calculation of an average template constituting an average of the biometric templates produced, - modification of a chosen image from a plurality of images into a modified image, the modification being adapted so that an error between the average template and a biometric template that would be produced by applying the processing to the modified image is less than an error between the average template and the biometric template that was produced by applying the processing to the chosen image.

[0008] On the one hand, the modified image condenses information from several images of the individual into a smaller size, thus reducing its likelihood of being noisy. On the other hand, applying the same processing to this image would produce a biometric template close to the average template, thereby constituting a reliable basis for comparison with reference templates.

[0009] The method according to the first aspect may further include the following optional features, taken alone or combined with each other where technically possible.

[0010] Preferably, the chosen image is such that the biometric template produced by applying the chosen image is closest to the average template among the biometric templates produced.

[0011] Preferably, the modification step is iterated until at least one of the following conditions is satisfied: - an iteration of the modification step produced a final image such that the error between the average template and a biometric template that would be produced by applying the processing to the final image is less than a single predefined value, - the number of iterations of the modification step has reached a predefined threshold.

[0012] Preferably the modification step is iterated as follows: h+i: - Ik is an image to be modified by an iteration k of the modification step, - Ik+i is a modified image produced by iteration k, - X is a predefined parameter, ■ n — ÔEk , °ù Ek is an error between the average template and a biometric template uk~ ôlk produced by applying the treatment to the image Ik.

[0013] Preferably, the processing is implemented by a convolutional neural network.

[0014] Preferably, the method includes a selection of the plurality of images in a sequence of images, the selection was carried out on the basis of an image quality criterion and on the basis of a temporal location criterion in the sequence of images.

[0015] Preferably, the selection includes: a) For each image in the image sequence, calculate an associated metric indicative of image quality, b) selection of an image from the image sequence whose associated metric indicates the best quality from a set of images in the sequence that have not yet been selected, c) for at least one image in the sequence other than the selected image that is located in a predefined temporal neighborhood of the selected image, degradation of the metric associated with the image, d) repeating steps b) and c) until at least one of the following conditions is met: - the number of implementations of step b) has reached a predefined threshold, - the metric of each image in the image sequence not yet selected has crossed a predefined quality threshold.

[0016] Preferably, the steps of the process are implemented by a first device, the process further comprising a step of transmission, in a communication network and to a second device, of a final image being derived from the modified image, the second device being configured to apply the processing to the final image, so as to produce a final biometric template relating to the individual.

[0017] It is also proposed, according to a second aspect, a computer-readable memory storing instructions executable by the computer for the execution of the steps of the process according to the first aspect.

[0018] A third aspect also proposes a device comprising a sensor configured to acquire a plurality of images showing an individual, and at least one processor configured to: - For each image that is part of the plurality of images, apply a processing to the image in order to produce a biometric template relating to the individual, - calculate an average template representing the average of each biometric template produced, - modify a chosen image from the plurality of images into a modified image, the modification being adapted so that an error between the average template and a biometric template produced by applying the processing to the modified image is less than an error between the average template and the biometric template having been produced by applying the processing to the chosen image. DESCRIPTION OF THE FIGURES

[0019] Other features, objectives and advantages of the invention will become apparent from the following description, which is purely illustrative and not limiting, and which should be read in conjunction with the accompanying drawings on which:

[0020] Fig. 1 schematically illustrates a biometric authentication system according to one embodiment of the invention.

[0021] Figure 2 is a flowchart of steps of a first process according to a mode of realization of the invention.

[0022] Fig. 3 details a flowchart showing how to carry out one of the steps shown in Fig. 2.

[0023] Figure 4 is a flowchart of steps of a second process according to a mode of realization of the invention.

[0024] Throughout the figures, similar elements bear identical references. DETAILED DESCRIPTION OF THE INVENTION

[0025] With reference to [Fig.1], a biometric authentication system comprises a first device 1 and a second device 2 adapted to communicate with each other via a communication network 3.

[0026] The first device 1 includes an image sensor 10, a data processing unit 12, a memory 14 and a communication interface 16 with the second device 2.

[0027] The image sensor 10 is for example adapted to acquire a video of an individual, that is to say a sequence of images showing an individual.

[0028] In the present text, an “image showing an individual” means an image which shows a part of that individual’s body, in particular a part of the individual’s body which makes it possible to distinguish the individual from another individual.

[0029] For example, the acquired images may show the individual's face, or at least a fingerprint of the individual.

[0030] The data processing unit 12 is configured to apply certain processing to the images acquired by the image sensor 10.

[0031] In particular, the processing unit 12 is configured to implement a convolutional neural network that takes as input an image I and produces from this input image a biometric template T (or "template") relating to the individual shown in the input image I. In the following, the processing applied by the convolutional neural network will be designated by the mathematical function CNN. Therefore:

[0032] T = CNN(l)

[0033] The processing unit 12 typically includes at least one processor to implement the aforementioned processing, the processor or processors being able to take any form (ASIC, FPGA, SoC, microcontroller, etc.).

[0034] The first device 1 further comprises a memory that stores a computer program comprising code instructions executable by the processor or each processor. When the program is executed by the processor or each processor, a process comprising the aforementioned operations is implemented.

[0035] Memory typically includes a volatile memory unit (e.g., RAM) and a non-volatile memory unit (flash, EEPROM, hard drive, SSD, etc.).

[0036] The communication interface 16 is adapted to allow the first device 1 to transmit data to the second device 2 via the communication network 3. This communication interface 16 can be of any type (wired or wireless radio), and communications via this communication network 3 can be governed by any communication protocol (Ethernet, Wi-Fi, Bluetooth, 3G / 4G / 5G cellular, etc.).

[0037] The first device 1 is typically intended for use at the entrance to a secure area, which only authorized individuals are permitted to enter. For example, the first device 1 may be part of, or coupled with, a gate constituting a physical barrier that prevents an unauthorized individual from entering the secure area, while allowing an authorized individual to enter. It will be seen later that the system is used for the biometric authentication of an individual, this biometric authentication having, for example, the purpose of determining whether this individual is authorized (in which case they will be allowed to enter the secure area) or not (in which case access to the secure area will be denied).

[0038] The second device 2 is distant from the first device 1.

[0039] The second device 2 includes a communication interface 20 with the first device 1, a processing unit 22 and a memory 24.

[0040] The communication interface 20 can be of the same type as the communication interface 16 of the first device 1.

[0041] The processing unit 22 is in particular configured to implement the same convolutional neural network with mathematical function CNN as the first device 1.

[0042] Memory 24 is intended to store a database comprising biometric templates or biometric images relating to various previously enrolled individuals.

[0043] With reference to [Fig.2], a method implemented in the first device 1 comprises the following steps.

[0044] An individual approaches the first device 1. In what follows, we will consider an example of implementation in which an individual wishes to enter a secure area as described previously.

[0045] In an acquisition step 100, the camera acquires a sequence of images showing the individual.

[0046] Typically, the image sequence comprises a number N of images acquired at a relatively high frequency, with N > 3.

[0047] The sequence of N images is stored in memory 14.

[0048] In a selection step 102, the processing unit 12 selects from the image sequence a plurality of images constituting a subset of images from this sequence. In other words, the plurality of images obtained at the end of this selection comprises a number of images K such that 1 < K < N.

[0049] The selection 102 of the plurality of images in the image sequence is carried out by the processing unit 12 on the basis of an image quality criterion and on the basis of a temporal location criterion in the image sequence.

[0050] In an embodiment represented in [Fig.3], selection 102 advantageously includes the following substeps.

[0051] For each image in the image sequence, the processing unit 12 calculates a metric associated with the image, the metric being indicative of the image quality (step 200). N metrics are therefore calculated in this step 200.

[0052] The calculated metric may be one of the following metrics, known to a person skilled in the art: • An indicative metric of a face pose of an individual shown in the image: in this case, the value of the metric depends on the orientation of the face relative to the image sensor 10; an image showing the individual's face from the front is considered to be of better quality than an image showing the individual's face from the side or even from the side. • An indicative metric of the resolution of the face of the individual shown in the image: the greater the distance between the eyes of the individual, counted in number of pixels in the image, the higher the quality of this image. • An indicative metric of image blur level: the blurrier the image, the lower the image quality. • A “biometric” metric, produced by a neural network different from the convolutional neural network discussed above.

[0053] The calculated metrics are stored in memory 16 in association with their respective images.

[0054] Next, the processing unit 12 selects the image from the image sequence that is associated with the metric indicating the best quality in the set of images of the sequence not yet selected (substep 202). At this stage, this set is the entire image sequence itself acquired by the image sensor 10; in other words, the image selected during this first implementation of substep 202 is the one with the highest quality within the image sequence. This image selected during this first implementation is conventionally called the "first image".

[0055] The processing unit 12 identifies each image in the image sequence that is in a temporal neighborhood of the image selected in substep 202 (i.e., the first image). An image in the sequence is considered a neighbor of the image selected in substep 202 if it lies within a predefined time interval that includes the selected image, for example, a time interval centered on that selected image. To illustrate this principle, consider a concrete example: if the radius of this time interval is set to 500 milliseconds, the images in the sequence that are at most 500 milliseconds ahead or behind the selected image can be considered neighbors of the selected image.

[0056] The processing unit 12 degrades the metric associated with at least one of the neighboring images (substep 204). By "degradation," we mean that the value of the metric is modified so as to decrease the quality indicated by the metric. It is understood that this degradation only concerns the calculated metric values, and not the images to which these metric values ​​refer; in other words, the images themselves are not degraded.

[0057] Preferably, the processing unit 12 degrades 204 the metric associated with any image adjacent to the image previously selected in substep 202. All degradations are, for example, shifts according to the same quality difference.

[0058] In step 206, the processing unit 12 checks whether at least one of the following output conditions is satisfied: • the number of implementations of sub-steps 202 and 204 reaches a predefined threshold Kmax, • The metric of each image in the image sequence that has not yet been selected indicates a quality lower than a predefined quality threshold.

[0059] If at least one of these conditions is met, selection step 102 is completed.

[0060] Otherwise, substeps 202, 204 and 206 are iterated as follows.

[0061] The processing unit 12 repeats substep 202, which consists of selecting an image of the image sequence which is associated with the metric indicating the best quality in the set of images in the sequence that have not yet been selected.

[0062] One difference from the previous implementation of this substep 202 is that the first image has been selected in the meantime; thus, at the In the second implementation of substep 202, the set of images in the sequence that have not yet been selected corresponds to the initial image sequence excluding the first image. A second image, different from the first, is therefore selected during this second implementation of substep 202.

[0063] Another difference from the previous implementation of this substep lies in the fact that, in the meantime, the metrics respectively associated with certain images (at least one of the images adjacent to the first image) have been degraded. This advantageously favors the selection, as the second image, of an image that is not temporally close to the first image.

[0064] The processing unit 12 repeats substep 204, which consists of degrading the metric associated with at least one of the images neighboring the last selected image, namely the second image. The same temporal neighborhood can be used.

[0065] This embodiment of the selection step 102 has the advantage of leading to the obtaining of a plurality of selected images which are not too close to each other temporally, and therefore not very redundant, while being of good quality.

[0066] Returning to [Fig. 2], the processing unit 12 passes each selected image as input to the convolutional neural network. The processing implemented by the convolutional neural network produces, based on each selected image, a corresponding biometric template relating to the individual shown in the image (step 104). Thus, based on the K selected images, the convolutional neural network produces K templates associated with step 104.

[0067] Each biometric template typically takes the form of a vector, for example of norm 1.

[0068] The biometric templates produced by the convolutional neural network are stored in memory 14.

[0069] In a step 106, the processing unit 12 then calculates an average template TM, which, as its name indicates, is an average of the K biometric templates produced previously. For example, the average template TM can be calculated as the sum of these K biometric templates divided by the sum of their respective norms (i.e., divided by K in vector boxes with a norm equal to 1).

[0070] The average template TM is stored in memory 16.

[0071] Subsequently, the processing unit 12 chooses one of the images from among the K images that have been selected, which is conventionally called the initial image Io.

[0072] The initial image Io can be any one of the K selected images on the basis of which the template TM was calculated.

[0073] However, in an advantageous embodiment, the initial image Io is chosen to be the image associated with the biometric template that is closest to the average template TM. To find this image, it suffices to calculate each distance between the average template and one of the K biometric templates produced, to identify the biometric template To that is at minimum distance from the average template among the K biometric templates produced, and then to identify the image Io that was passed as input to the convolutional neural network with mathematical function CNN to produce the biometric template To that is at this minimum distance (To = CNN(Io)).

[0074] Subsequently, the processing unit 12 implements a loop comprising one or more iterations, this loop producing a final image If from the initial image Io.

[0075] We will first describe an initial iteration of this loop (by convention of index zero), before describing in a more general way an iteration of index k of this loop with respect to a previous iteration of index k-1.

[0076] The initial iteration of the loop includes a modification step 110 during which the processing unit 12 modifies the initial image Io into a modified image L. The modification 110 implemented is adapted to satisfy the following condition: an error Ei between the average template TM and a biometric template Ti that would be produced by applying the processing to the modified image L is less than an error Eo between the average template TM and the biometric template To that was produced by applying the processing to the initial image Io.

[0077] It should be noted here that this modification step does not necessarily require the prior calculation of the biometric template Ti, which would be produced by applying the processing to the modified image h, nor of the error Ei (hence the use of the conditional tense). However, such a calculation can be implemented in practice to verify whether an image modification is covered by the modification performed in step 110.

[0078] In one embodiment, the modification 110 of the image Io into the modified image L is carried out in the following manner.

[0079] Processing unit 12 calculates the following gradient:

[0080] n - siu

[0081] where Eo is an error between the average template and the biometric template To produced by applying the CNN processing to the image I0 (that is to say To = CNN(I0)).

[0082] The error Eo is, for example, a quadratic error, of the following form:

[0083] Eq=(T0-TM)2

[0084] Next, the processing unit 12 calculates the modified image h by applying the following formula:

[0085]

[0086] where X is a predefined parameter.

[0087] This embodiment of the modification step 110 is inspired by the gradient descent method, traditionally used to update the parameters of a neural network in a way that minimizes loss. However, in the present method, it is not the parameters of the convolutional neural network with mathematical function CNN that are to be modified, but an image that has been passed as input to this convolutional neural network (in this case, image Io), which is unconventional.

[0088] The initial iteration of the loop then includes a check step 112 during which the processing unit 12 checks whether one of the following output conditions is satisfied.

[0089] A first condition is satisfied when the error between the average template TM and the biometric template Ti produced by applying the processing to the image L is less than a predefined threshold E, otherwise when:

[0090] E{ = (TrTM)2 <E

[0091] Thus, to know if this first condition is satisfied, the processing unit 12 calculates Ti = CNN(Ii), when this calculation has not already been done during the step of modifying the image Io into the modified image L (this calculation is in particular not done in the embodiment described above inspired by the gradient descent method).

[0092] A second condition is satisfied when the number of iterations of the modification step 110 implemented reaches a predefined threshold. At this stage, the modification step has only been implemented once.

[0093] If the first condition is met or if the second condition is met, then the loop terminates. The final If image is then the modified Ip image

[0094] Otherwise, a new iteration of the loop (of index 1) is implemented, this new iteration taking as input the image L instead of the initial image Io.

[0095] Let us consider more generally an iteration of index k of the loop, with k^0. This iteration of index k applies to the image Ik, obtained during a previous iteration of index k-1, the modification of step 110 described previously, so as to obtain a new modified image Ik+1.

[0096] In the embodiment described above, which is based on the gradient descent method, we thus have at iteration k:

[0097] Ik+l = Ik-^Dk

[0098] where:

[0099] n _ SE, "k- 81 k

[0100] Ek=(Tk-TM)2

[0101] Modification 110 implemented at iteration k is adapted so that the error Ek+i between the average template TM and the biometric template Tk+i that would be produced by applying the processing to the modified image Ik+[ is less than the error Ek between the average template and the biometric template that was produced by applying the processing to the image Ik.

[0102] The final image If obtained at the output of the loop is the modified image produced during the last implementation of the modification step 110. As previously stated, the final image If can perfectly well be the image Ii (in the case where a single iteration of the loop was sufficient to lead to the satisfaction of the first or second condition), or be an image resulting from one or more successive modifications of this image h (in the case where several iterations of the loop had to be executed to satisfy the first or second condition).

[0103] Subsequently, the final image If is transmitted to the second device 2 via the communication interface (step 114).

[0104] The final image If has a double advantage: it is a small data file whose transmission consumes little bandwidth on the communication network 3, while compiling visual information from the K images selected in the video sequence. This final image If is less likely to be noisy than one of the original images of the acquired image sequence.

[0105] With reference to [Fig.4], the second device 2 implements a process comprising the following steps.

[0106] The second device 2 receives the final image If via its communication interface (step 300).

[0107] The processing unit 22 of the second device 2 passes the final image If as input to the same convolutional neural network with mathematical function CNN as that implemented by the first device 1 (step 302). In other words, the processing implemented in step 302 on the side of the second device 2 is identical to each processing implemented in step 104 on the side of the first device 1. From the final image If, ​​the convolutional neural network produces in step 302 a biometric template Tf = CNN(If), which is conventionally called the "final" biometric template.

[0108] The processing unit 12 can then compare the final biometric template Tf with the reference templates stored in the memory database 24, in order to determine whether the final biometric template Tf corresponds to one of them (this comparison processing is known in English as "matching"). • If yes, this means that the individual has been previously enrolled in the database; the individual is therefore an authorized user, and is given authorization to access the secure area (positive authentication result). • If the answer is no, this means that the individual is not an authorized user; access to the security zone is denied (negative authentication result).

[0109] Due to the way in which the final image If was produced, the final biometric template Tf is close to the average template TM. This property allows the second device 2 to implement reliable matching. Thus, a dual objective was achieved: the biometric authentication consumed very little bandwidth on the communication network 3 between the first device 1 and the second device 2 (only one information-rich image was transmitted over this network), without compromising the proper functioning of the template matching.

[0110] The method implemented by the first device 1 can also be combined with a method implemented by the second device 2 to enroll the individual in the database of memory 24 (in which case the comparison / matching step is simply replaced by a memorization of the final template Tf and / or the final image If in the database).

[0111] Of course, the methods described above in relation to the attached figures are only embodiments, not limiting ones. These methods may be subject to variations.

[0112] Although advantageous, the use of a convolutional neural network to produce biometric templates from images is not necessary. Other processing methods can be implemented to produce biometric templates from images.

[0113] It is further understood that the acquisition step 100 and the selection step 102 are optional preliminary steps for obtaining the final image If. The selection step 102 is optional, and the plurality of K images on the basis of which the final image is obtained does not necessarily come from a sequence of images acquired by an image sensor 10.

Claims

Claims

1. A computer-implemented image generation method, the method comprising: - For each image forming part of a plurality of images showing an individual, applying (104) to the image a processing operation so as to produce a biometric template relating to the individual, - calculating (106) an average template (TM) constituting an average of the biometric templates produced, - modifying (110) a selected image (Io) from the plurality of images into a modified image (IJ), the modification (110) being adapted so that an error between the average template (TM) and a biometric template that would be produced by applying the processing to the modified image is less than an error between the average template (TM) and the biometric template having been produced by applying the processing to the selected image (Io).

2. Method according to the preceding claim, in which the chosen image (Io) is such that the biometric template produced by applying the processing to the chosen image (Io) is closest to the average template (TM) among the biometric templates produced.

3. Method according to one of the preceding claims, in which the modification step (110) is iterated until at least one of the following conditions is satisfied: - an iteration of the modification step (110) has produced a final image (If) such that an error between the average template (TM) and a biometric template which would be produced by applying the processing to the final image (If) is less than a predefined one, - the number of iterations of the modification step (110) has reached a predefined threshold.

4. Method according to one of the preceding claims, in which the modification step (110) is iterated as follows: Ik+i = Ik-A*Dk^ - Ik is an image to be modified by an iteration k of the modification step (110), - Ik+i is a modified image produced by the iteration k, - / . is a predefined parameter, " n ■> °ù Ek is an error between the average template (TM) and a uk~ &Ik biometric template produced by applying the processing to the image Ik.

5. Method according to one of the preceding claims, in which the processing is implemented by a convolutional neural network.

6. Method according to one of the preceding claims, further comprising: - a selection (102) of the plurality of images in a sequence of images, the selection being carried out on the basis of an image quality criterion and on the basis of a temporal location criterion in the sequence of images, the selection (102) comprising: a) for each image of the sequence of images, calculating (200) an associated metric indicative of a quality of the image, b) selecting (202) an image of the sequence of images whose associated metric indicates the best quality in a set of images of the sequence not yet having been selected, c) for at least one image of the sequence other than the selected image located in a predefined temporal neighborhood of the selected image, degrading (204) the metric associated with the image,d) repeating steps b) and c) until at least one of the following conditions is satisfied: - the number of implementations of step b) has reached a predefined threshold, - the metric of each image in the image sequence not yet selected has crossed a predefined quality threshold.,

7. Method according to one of the preceding claims, the steps of which are implemented by a first device (1), the method further comprising a step of - transmission (114), in a communication network (3) and to a second device (2), of a final image (If) originating from the modified image, the second device (2) being configured to apply the processing to the final image, so as to produce a final biometric template (Tf) relating to the individual.

8. Computer-readable memory (14) storing computer-executable instructions for carrying out the steps of the method according to one of the preceding claims.

9. Device (1) comprising a sensor (10) configured to acquire a plurality of images showing an individual, and at least one processor (12) configured to: - for each image forming part of the plurality of images, apply to the image a processing so as to produce a biometric template relating to the individual, - calculate an average template constituting an average of each biometric template produced, - modify a chosen image (Io) in the plurality of images into a modified image (h), the modification being adapted so that an error between the average template and a biometric template produced by applying the processing to the modified image is less than an error between the average template and the biometric template having been produced by applying the processing to the chosen image.

Images