Method for generating an augmented image and associated apparatus

Abstract

The invention relates to a method for generating an enhanced image and an associated apparatus. A method comprises, for each image forming part of a plurality of images of a display individual, applying (104) a process to the image in order to produce a biometric template relating to the individual, calculating (106) an average template constituting an average level of the produced biometric templates, modifying (110) a selected image (I0) from the plurality of images into a modified image (I1), the modification (110) being adapted so that an error between the average template (TM) and a biometric template produced by applying the process to the modified image is less than an error between the average template (TM) and the biometric template that has been produced by applying the process to the selected image (I0).

Description

Technical Field

[0001] This invention relates to the field of biometrics.

[0002] More specifically, the present invention relates to a method comprising generating an enhanced image from multiple images of an individual. Background Technology

[0003] Known biometric authentication systems include two devices: a first device including an image sensor for acquiring images, and a second device for generating authentication results, with the first and second devices communicating via a communication network.

[0004] The first method implemented in this type of system is as follows: An image sensor acquires and displays an image of the individual. This image is transmitted to a second device via a communication network. The second device generates a biometric template based on the transmitted image. This biometric template can then be compared with reference templates stored in a database. If the calculated biometric template matches one of the reference templates, the individual is considered an authorized user. However, if the acquired image is noisy, especially if the image is blurry or does not accurately display the distinctive features of the user's body, this first method may lead to erroneous results.

[0005] To overcome this problem, a second method has been proposed, in which the image sensor acquires not just one, but multiple images, and the processing implemented in the first method is applied to each image. However, this second method has several drawbacks: first, it consumes more bandwidth compared to the first method (specifically, transmitting multiple images, not just one, over the communication network); second, it requires a greater computational load compared to the first method. Summary of the Invention

[0006] One object of the present invention is to generate individual-related data, based on which biometric authentication can be reliably performed and with low bandwidth consumption during data transmission in communication networks.

[0007] Therefore, according to the first aspect, a computer-implemented method is provided, the method comprising:

[0008] - For each image that forms part of a set of multiple images showing an individual, the image is processed to generate a biometric template associated with that individual;

[0009] - Calculate the average template that constitutes the average level of the generated biometric template;

[0010] - Modify a selected image from the plurality of images into a modified image, the modification being adapted such that the error between the average template and another biometric template generated by applying the processing to the modified image is less than the error between the average template and the biometric template generated by applying the processing to the selected image.

[0011] First, the modified image compresses information from multiple images of the displayed individual into a smaller size, reducing the likelihood of image noise. Second, applying the same processing to the image will produce a biometric template that approximates the average template, thus providing a reliable basis for comparison with the reference template.

[0012] The method according to the first aspect may further include the following optional features, which may be used individually or in combination with each other when technically feasible.

[0013] Preferably, the selected image is one that makes the biometric template generated by applying the selected image as close as possible to the average template among the generated biometric templates.

[0014] Preferably, the modification steps are iterated until at least one of the following conditions is met:

[0015] - The iteration of this modification step has produced a final image such that the error between the average template and the biometric template produced by applying the processing to the final image is below a predefined threshold;

[0016] - The number of iterations for this modification step has reached a predefined threshold.

[0017] Preferably, the modification step is iterated as follows:

[0018] I k+1 =I k -λ*D k

[0019] in:

[0020] -I k It is the image that is modified through the k-th iteration of this modification step;

[0021] -I k+1 It is the modified image generated through the k-th iteration;

[0022] -λ is a predefined parameter;

[0023] - Among them, E k Is the average template and the image I k Errors between biometric templates generated by applying this processing.

[0024] Preferably, this processing is implemented using a convolutional neural network.

[0025] Preferably, the method includes selecting the plurality of images from an image sequence, for example, a video originating from an individual.

[0026] Preferably, the multiple images are derived from images acquired by a single image sensor.

[0027] Preferably, the selection is based on image quality standards and time position standards in the image sequence.

[0028] Preferably, the selection includes:

[0029] a) For each image from the image sequence, calculate an associated metric indicating the quality of that image;

[0030] b) Select images from the image sequence, the associated metric of which indicates the best quality from a set of images in the sequence that have not yet been selected;

[0031] c) For at least one image from the sequence that is located in a predefined temporal neighborhood of the selected image, other than the selected image, degrade the metric associated with that image;

[0032] d) Repeat steps b) and c) until at least one of the following conditions is met:

[0033] - The number of times step b) has been implemented has reached the predefined threshold;

[0034] - The metric for each image from this image sequence that has not yet been selected has exceeded a predefined quality threshold.

[0035] Preferably, the steps of the method are performed by a first device, and the method further includes the following steps: transmitting a final image derived from the modified image to a second device in a communication network, the second device being configured to apply the processing to the final image to generate a final biometric template associated with the individual.

[0036] According to the second aspect, a computer-readable storage medium is also provided, which stores instructions that can be executed by the computer to perform the steps of the method according to the first aspect.

[0037] According to a third aspect, an apparatus is also provided, the apparatus comprising: a sensor configured to acquire and display multiple images of an individual; and at least one processor configured to:

[0038] - For each image that forms part of the multiple images, apply processing to that image to generate a biometric template associated with that individual;

[0039] - Calculate the average template of the average level of each biometric template generated;

[0040] - Modify a selected image from the plurality of images into a modified image, the modification being adapted such that the error between the average template and the biometric template generated by applying the processing to the modified image is less than the error between the average template and the biometric template generated by applying the processing to the selected image. Attached Figure Description

[0041] Further features, objects, and advantages of the invention will become apparent from the following description, which is purely illustrative and not restrictive, and should be read in conjunction with the accompanying drawings, in which:

[0042] Figure 1 A biometric authentication system according to an embodiment of the present invention is illustrated schematically.

[0043] Figure 2 This is a flowchart of the steps of a first method according to an embodiment of the present invention.

[0044] Figure 3 The details are explained in detail. Figure 2 The flowchart shows one embodiment of one of the steps.

[0045] Figure 4 This is a flowchart of the steps of a second method according to an embodiment of the present invention.

[0046] In all the accompanying drawings, similar elements have the same reference numerals. Detailed Implementation

[0047] refer to Figure 1 The biometric authentication system includes a first device 1 and a second device 2 designed to communicate with each other via a communication network 3.

[0048] The first device 1 includes an image sensor 10, a data processing unit 12, a memory 14, and a communication interface 16 with the second device 2.

[0049] Image sensor 10 is designed, for example, to acquire video of an individual, which is a sequence of images of the individual.

[0050] In this article, “displaying an image of an individual” means displaying an image of a part of that individual’s body, in particular a part of an individual’s body that can distinguish that individual from another individual.

[0051] For example, the captured images may show an individual's face, or at least their fingerprints.

[0052] The data processing unit 12 is configured to apply certain processing operations to the image acquired by the image sensor 10.

[0053] Specifically, processing unit 12 is configured to implement a convolutional neural network that takes image I as input and generates a biometric template T from the input image that is related to the individual displayed in the input image I. Hereinafter, the processing applied by the convolutional neural network will be referred to as the CNN mathematical function. Therefore:

[0054] T = CNN(I)

[0055] The processing unit 12 typically includes at least one processor for performing the aforementioned processing operations, and the processor or each processor may take any form (ASIC, FPGA, SoC, microcontroller, etc.).

[0056] The first device 1 further includes a memory storing a computer program, the computer program including code instructions executable by the processor or each processor. When the program is executed by the processor or each processor, a method including the aforementioned processing operations is implemented.

[0057] This memory typically includes volatile memory cells (such as RAM) and non-volatile memory cells (flash memory, EEPROM, hard disk, SSD, etc.).

[0058] The communication interface 16 is designed to enable the first device 1 to transmit data to the second device 2 via the communication network 3. The communication interface 16 can be of any type (wired or wireless radio), and communication via the communication network 3 can be controlled by any communication protocol (Ethernet, Wi-Fi, Bluetooth, 3G / 4G / 5G cellular, etc.).

[0059] The first device 1 is typically intended for use at the entrance to a secure area, where only authorized individuals are permitted entry. For example, the first device 1 may form part of or be coupled to a door that provides a physical barrier preventing unauthorized individuals from entering the secure area while allowing authorized individuals to enter. As will be seen later, the system is used to perform biometric authentication of an individual, the purpose of which is, for example, to determine whether the individual is authorized (in which case, the individual will be allowed entry into the secure area) or unauthorized (in which case, the individual will be denied entry into the secure area).

[0060] According to a variant (not shown), the first device 1 includes multiple image sensors designed to acquire multiple images of an individual.

[0061] For example, this allows images of individuals to be taken from different angles. The second device 2 is located away from the first device 1.

[0062] The second device 2 includes a communication interface 20 with the first device 1, a processing unit 22, and a memory 24.

[0063] The communication interface 20 can be of the same type as the communication interface 16 of the first device 1.

[0064] Specifically, the processing unit 22 is configured to implement the same convolutional neural network (CNN) mathematical functions as the first device 1.

[0065] The memory 24 is intended to store a database that includes biometric templates or biometric images associated with various previously registered individuals.

[0066] refer to Figure 2 The method of implementing the first device 1 includes the following steps.

[0067] An individual approaches the first device 1. In the following, an exemplary implementation will be considered, in which the individual wishes to enter the security area as described above.

[0068] In acquisition step 100, the camera acquires, for example, a sequence of images of an individual.

[0069] Typically, an image sequence consists of N images acquired at a relatively high frequency, where N ≥ 3.

[0070] A sequence containing N images is stored in memory 14.

[0071] According to one variation, multiple sensors acquire and display images of an individual. The image sequence comprises images of the individual taken by various different sensors at various time points. In selection step 102, processing unit 12 selects multiple images from the image sequence that constitute a subset of the images in the sequence. In other words, the multiple images obtained from this selection comprise K images, such that 1 ≤ K ≤ N.

[0072] The multiple images are selected 102 from the image sequence by the processing unit 12 based on image quality criteria and time position criteria in the image sequence.

[0073] exist Figure 3 In one embodiment shown, selection 102 advantageously includes the following sub-steps.

[0074] For each image in the image sequence, processing unit 12 calculates a metric associated with that image, which indicates the quality of the image (step 200). Therefore, N metrics are calculated in step 200.

[0075] The calculated metric may be one of the following metrics known to those skilled in the art:

[0076] • A metric indicating the position of an individual's face displayed in an image: In this case, the value of the metric depends on the orientation of the face relative to the image sensor 10; an image showing an individual's face from the front is considered to be of better quality than an image showing an individual's face from a certain angle or even from the side.

[0077] • A measure indicating the resolution of an individual's face displayed in an image: the greater the distance between an individual's eyes (in pixels in the image), the higher the quality of the image.

[0078] • A measure indicating the degree of blur in an image: the blurrier the image, the lower its quality.

[0079] • Biometrics generated by neural networks that are different from the convolutional neural networks mentioned above.

[0080] The calculated metric is stored in memory 16 in association with its corresponding image.

[0081] Next, processing unit 12 selects an image from the image sequence associated with a metric indicating the best quality from a set of images that have not yet been selected in the sequence (sub-step 202). At this stage, this set is the entire image sequence acquired by image sensor 10; in other words, the image selected in this first embodiment of sub-step 202 is the image with the highest quality within the image sequence. By convention, this image selected during this first embodiment is referred to as the "first image".

[0082] Processing unit 12 identifies each image in the image sequence that is within the temporal neighborhood of the image selected in sub-step 202 (i.e., the first image). If an image in the sequence is located within a predefined time range that includes the selected image, such as a time range centered on the selected image, then that image is considered to be within the neighborhood of the image selected in sub-step 202. To illustrate this principle with a concrete example: if the radius of this time range is chosen to be 500 milliseconds, then those images in the sequence that are at most 500 milliseconds earlier or later than the selected image can be considered to be images within (or near) the neighborhood of the selected image.

[0083] Processing unit 12 downgrades the metric associated with at least one neighboring image in the neighboring images (sub-step 204). "Downgrade" means modifying the value of the metric to reduce the quality indicated by the metric. It should be understood that this downgrade only affects the calculated metric values ​​and does not affect the images associated with these metric values; that is, the images themselves are not downgraded.

[0084] Preferably, processing unit 12 downgrades 204 any metrics associated with any neighboring images previously selected in sub-step 202. For example, all downgrades are transformations based on the same quality difference.

[0085] In step 206, processing unit 12 checks whether at least one of the following output conditions is met:

[0086] • The number of times sub-steps 202 and 204 are implemented reaches the predefined threshold Kmax;

[0087] • The metric for each unselected image from the image sequence indicates quality below a predefined quality threshold.

[0088] If at least one of these conditions is met, then select step 102 to end.

[0089] Otherwise, proceed with the following iterative sub-steps 202, 204, and 206.

[0090] Processing unit 12 repeats sub-step 202, which involves selecting from the image sequence an image associated with a metric indicating the best quality from a set of images that have not yet been selected in the sequence.

[0091] One difference from the aforementioned implementation of sub-step 202 is that a first image is selected simultaneously; therefore, in the second implementation of sub-step 202, the set of images from the sequence that have not yet been selected corresponds to the initial image sequence other than the first image. Thus, in this second implementation of sub-step 202, a second image different from the first image is selected.

[0092] Another difference from the aforementioned implementation of this sub-step is that the metrics associated with certain images (at least one of the images adjacent to the first image) have been simultaneously downgraded. Advantageously, this facilitates the selection of images that are not temporally close to the first image as the second image.

[0093] Processing unit 12 repeats sub-step 204, which involves downgrading the metric associated with at least one of the neighboring previously selected images (i.e., the second image). The same temporal neighborhood can be used.

[0094] The advantage of this embodiment of step 102 is that it results in obtaining multiple selected images that are not too close to each other in time and are therefore not very redundant, while still having good quality.

[0095] Return to Figure 2The processing unit 12 passes each selected image as input to the convolutional neural network. The processing performed by the convolutional neural network generates a corresponding biometric template associated with the individual displayed in the image based on each selected image (step 104). Therefore, in step 104, the convolutional neural network generates K associated templates based on K selected images.

[0096] Each biometric template is typically in vector form, such as a vector with norm 1.

[0097] The biometric templates generated by the convolutional neural network are stored in memory 14.

[0098] In step 106, processing unit 12 then calculates the average template TM, which, as its name suggests, constitutes the average level of the previously generated K biometric templates. For example, the average template TM can be calculated as the sum of these K biometric templates divided by the sum of their respective norms (i.e., divided by K in the case of a norm 1 vector).

[0099] The average template TM is stored in memory 16.

[0100] Next, the processing unit 12 selects one of the K images that have been selected, which is conventionally referred to as the initial image I0.

[0101] The initial image I0 can be any image from among these K selected images, based on which the template TM has been calculated.

[0102] However, in an advantageous embodiment, the image associated with the biometric template closest to the average template TM is selected as the initial image I0. To find this image, it is only necessary to calculate each distance between the average template and one of the resulting K biometric templates to identify the biometric template T0 with the smallest distance from the average template among the resulting K biometric templates, and then identify the image I0 as input to the mathematical function of the convolutional neural network CNN to produce the biometric template T0 at that minimum distance (T0 = CNN(I0)).

[0103] Next, the processing unit 12 implements a loop including one or more iterations, which generates the final image I from the initial image I0. f .

[0104] Before describing the iteration k of the loop relative to the previous iteration with index k-1 in a more general way, the initial iteration of the loop (conventionally indexed to zero) will be described first.

[0105] The initial iteration of the loop includes a modification step 110, in which the processing unit 12 modifies the initial image I0 into a modified image I1. The modification 110 is designed to satisfy the condition that the error E1 between the average template TM and the biometric template T1 generated by applying processing to the modified image I1 is less than the error E0 between the average template TM and the biometric template T0 generated by applying processing to the initial image I0.

[0106] It should be noted here that this modification step does not necessarily require pre-calculating the biometric template T1 generated by applying processing to the modified image I1 or pre-calculating the error E1. However, such calculation can actually be implemented to check whether the image modification is overridden by the modification performed in step 110.

[0107] Even if the biometric template T1 is not calculated according to the steps of this method, the modification step 110 ensures that the conditions regarding errors E1 and E2 are met, namely, the error E1 between the average template TM and the biometric template T1 generated by applying processing to the modified image I1 is less than the error E0 between the average template TM and the biometric template T0 generated by applying processing to the initial image I0.

[0108] In one embodiment, the image I0 is modified to the modified image I1 as follows.

[0109] Processing unit 12 calculates the following gradients:

[0110]

[0111] Here, E0 is the error between the average template and the biometric template T0 generated by applying CNN processing to image I0 (i.e., T0 = CNN(I0)).

[0112] For example, the error E0 has the following form:

[0113] E0 = (T0 - TM) 2

[0114] Next, the processing unit 12 calculates the modified image I1 by applying the following formula:

[0115] I1=I0-λ*D0

[0116] Where λ is a predefined parameter.

[0117] This embodiment of modifying step 110 is based on gradient descent, which is traditionally used to update the parameters of a neural network in a way that minimizes the loss. However, in this method, what needs to be modified is not the parameters of the mathematical function of the convolutional neural network (CNN), but the image (in this case, image I0) that has already been passed as input to the CNN, which is not conventional.

[0118] Then, the initial iteration of the loop includes a check step 112, in which the processing unit 12 checks whether one of the following output conditions is met.

[0119] The first condition is met when the error between the average template TM and the biometric template T1 generated by applying processing to image I1 is less than a predefined threshold E, or when the following occurs:

[0120] E1 = (T1 - TM) 2 <E

[0121] Therefore, in order to know whether this first condition is met, the processing unit 12 performs this calculation when the calculation T1 = CNN(I1) has not been performed in the step of modifying image I0 into the modified image I1 (in particular, this calculation is not performed in the embodiment described above based on gradient descent).

[0122] The second condition is met when the number of iterations of the modification step 110 reaches a predefined threshold. At this stage, the modification step is implemented only once.

[0123] The loop ends if either the first condition or the second condition is met. Final image I f This is the modified image I1.

[0124] Otherwise, a new iteration of the loop (index 1) is implemented, which takes image I1 instead of the initial image I0 as input.

[0125] More generally, consider the iteration with index k in the loop, where k ≠ 0. This iteration with index k is applied to the image I obtained in the previous iteration with index k-1 (i.e., the modification of step 110 above). k In order to obtain a new modified image I k+1 .

[0126] In the gradient descent-inspired embodiment described above, therefore at the iteration with index k:

[0127] I k+1 =I k -λ*D k

[0128] in:

[0129]

[0130] E k =(T k -TM) 2

[0131] The modification 110 implemented in the iteration at index k is designed to make the average template TM equal to the modified image I. k+1 Biometric template T generated by application processing k+1 The error E between k+1 Less than the average template and has been passed through image I k Error E between biometric templates generated by application processing k .

[0132] The final image I obtained as the output of the loop f This is the modified image generated in the previous implementation of step 110. As mentioned above, the final image I f It is likely that the image is I1 (in cases where a single loop iteration is sufficient to satisfy either the first or second condition), or the final image may be an image generated by one or more consecutive modifications to the image I1 (in cases where multiple loop iterations must be run to satisfy either the first or second condition).

[0133] Final Image I f Then it is transmitted to the second device 2 via the communication interface (step 114).

[0134] Final Image I f It has two advantages: the final image is a small data item, and its transmission 114 consumes almost no bandwidth on the communication network 3, while combining visual information from K images selected from the video sequence. This final image I f It is less likely to have noise compared to one of the original images in the acquired image sequence.

[0135] refer to Figure 4 The second device 2 implements a method including the following steps.

[0136] The second device 2 receives the final image I via its communication interface. f (Step 300)

[0137] The processing unit 22 of the second device 2 will process the final image I f The input is passed to the same convolutional neural network (CNN) mathematical function implemented by the first device 1 (step 302). In other words, the processing implemented on the second device 2 side in step 302 is the same as each processing operation implemented on the first device 1 side in step 104. In step 302, the convolutional neural network inputs the final image I...f Generate biometric template T f =CNN(I f This biometric template is conventionally referred to as the "final" biometric template.

[0138] Processing unit 12 can then process the final biometric template T f The final biometric template T is compared with a reference template stored in memory database 24 to determine the biometric template T. f Does it match any of these reference templates?

[0139] If so, it means that the individual has previously registered in the database; therefore, the individual is an authorized user and is authorized to access the secure area (positive authentication result).

[0140] If not, it means the individual is not an authorized user and is denied access to the secure area (negative authentication result).

[0141] Due to the final image I f The generation method, the final biometric template T f The approximate average template TM. This proximity allows the second device 2 to perform a still reliable matching. Thus, two goals are achieved: the performed biometric authentication consumes almost no bandwidth on the communication network 3 between the first device 1 and the second device 2 (only one information-rich image is transmitted through this network), and this is achieved without compromising the normal operation of the template matching operation.

[0142] The method implemented by the first device 1 can also be combined with the method implemented by the second device 2 to register individuals in the database in the memory 24 (in this case, the comparison / matching step is simply replaced by storing the final template T in the database). f and / or final image I f ).

[0143] Of course, the methods described above with reference to the accompanying drawings are merely non-limiting embodiments. These methods may have variations.

[0144] While using convolutional neural networks to generate biometric templates from images is advantageous, it is not mandatory. Other processing operations can be implemented to generate biometric templates from images.

[0145] It should be further understood that acquisition step 100 and selection step 102 are used to obtain the final image I. f Optional preparatory steps. Step 102 is optional, and the K images on which the final image is based do not necessarily come from the image sequence acquired by the image sensor 10.

Claims

1. A computer-implemented method for processing images containing individual-related biometric data, the method comprising the following steps: - For each image that forms part of a plurality of images showing an individual, the image is processed by (104) to generate a biometric template associated with that individual; - Calculate (106) the average template of the generated biometric templates; - A selected image from the plurality of images is modified (110) into a modified image, the modification (110) being adapted such that the error between the average template and another biometric template generated by applying the processing to the modified image is less than the error between the average template and the biometric template generated by applying the processing to the selected image. The iteration of the modification (110) is as follows: in: - I k It is the image that is modified by the kth iteration of the modification (110) step; - I k+1 It is the modified image generated through the k-th iteration; - λ is a predefined parameter; - , of which E k Is the average template and the image I k Errors between biometric templates generated by applying this processing.

2. The method according to claim 1, wherein, The selected image is chosen so that the biometric template generated by applying the selected image is closest to the average template among the generated biometric templates.

3. The method according to claim 1, wherein, The steps of iteratively modifying (110) continue until at least one of the following conditions is met: - The iteration of the modification (110) step has produced a final image such that the error between the average template and the biometric template produced by applying the processing to the final image is below a predefined threshold; - The number of iterations of the modified (110) step has reached the predefined threshold.

4. The method according to claim 1, wherein, This processing is implemented using a convolutional neural network.

5. The method of claim 1, further comprising: - Select (102) the plurality of images from the image sequence, the selection being based on image quality criteria and time location criteria in the image sequence.

6. The method according to claim 5, wherein, The selection (102) includes the following steps: a) For each image from the image sequence, calculate (200) an associated metric indicating the quality of that image; b) Select (202) images from the image sequence, the associated metric of which indicates the best quality from a set of images in the sequence that have not yet been selected; c) For at least one image from the sequence that is located in a predefined temporal neighborhood of the selected image, other than the selected image, degrade the metric associated with that image (204). d) Repeat steps b) and c) until at least one of the following conditions is met: - The number of times step b) has been implemented has reached the predefined threshold; - The metric for each unselected image from this image sequence has exceeded a predefined quality threshold.

7. The method according to claim 1, wherein the steps of the method are performed by the first device (1), and the method further includes the following steps: - Transmit (114) a final image derived from the modified image to a second device (2) in the communication network (3), the second device (2) being configured to apply the processing to the final image in order to generate a final biometric template associated with the individual.

8. A computer-readable storage medium (14) storing instructions that can be executed by the computer to perform the steps of the method according to any one of claims 1 to 7.

9. An apparatus (1) for processing images including individual-related biometric data, comprising: Sensor (10), which is configured to acquire and display multiple images of an individual; and at least one processor (12), which is configured to: - For each image that forms part of the multiple images, apply processing to that image to generate a biometric template associated with that individual; - Calculate the average template of the average level of each biometric template generated; and - A selected image from the plurality of images is modified (110) into a modified image, the modification being adapted such that the error between the average template and the biometric template generated by applying the processing to the modified image is less than the error between the average template and the biometric template generated by applying the processing to the selected image. The iteration of this modification (110) is as follows: in: - I k It is the image that will be modified through the k-th iteration of this modification (110); - I k+1 It is the modified image generated through the k-th iteration; - λ is a predefined parameter; - , of which E k Is the average template and the image I k Errors between biometric templates generated by applying this processing.

Images