Method, device and system for determining a facial image with improved image quality

Abstract

The invention relates to a method having the following steps which are carried out by a computer: receiving a plurality of facial images having at least partially different brightness levels; determining a parameter value for each of a plurality of image parameters for each of the plurality of facial images; assigning each of the plurality of facial images to one of a plurality of groups of persons if, for each of the plurality of image parameters, the parameter value thereof lies within a respective image parameter value range that is specific to the group of persons; selecting the facial images in the group of persons with the largest number of assigned facial images; and selecting, from the facial images in the selected group of persons, the facial image which has the highest parameter value for a selected image parameter of the plurality of image parameters as the optimal facial image.

Description

[0001] Method, apparatus and system for determining a facial image with improved image quality

[0002] The invention relates to a method, a device and a system for determining a facial image with improved image quality, as well as a computer program product comprising instructions that cause a correspondingly designed device to carry out the method for determining a facial image with improved image quality.

[0003] Facial images are often used to authenticate individuals, as passport photos, or for identification documents. In particular, facial images are stored as biometric data in electronic passports. For this purpose, the facial images must meet specified requirements regarding facial orientation, the recognizability of certain facial features, and, in particular, the representation of skin tones.

[0004] Such facial images are captured, for example, by individuals specially trained to capture facial images, such as photographers or clerks at a public location. This person is usually able to determine whether a facial image meets the requirements and, if necessary, make subsequent changes or capture a new image with modified settings. Such requirements are specified, for example, by the ICAO DOC 93O3 standard.

[0005] Recently, there has been a trend toward allowing people to authenticate themselves or take facial images for ID documents and passport photos independently. The presence of a trained person is no longer required. However, this also means that it is not possible to independently determine whether an image meets the specified requirements.

[0006] Instead of relying on a trained human, algorithms are used to assess whether a facial image meets the requirements. The use of deep learning-based algorithms has already led to significant progress in facial recognition and the determination of facial orientation.

[0007] However, an algorithm-based assessment of image quality, particularly the realistic reproduction of skin tones, has proven difficult, as skin tones can vary considerably between individuals. Currently, it is common for images taken independently without a trained person and controlled by an algorithm to not have the necessary image quality to meet biometric requirements, particularly regarding the quality of skin tones. These images often exhibit inappropriate contrast values ​​and / or inappropriate facial brightness values.

[0008] Against this background, it is an object of the invention to provide a facial image with improved image quality, in particular with regard to skin tones.

[0009] The object is achieved by a method according to independent claim 1. Furthermore, a device according to claim 11, a computer program product according to claim 12 and a system according to claim 13 are provided.

[0010] A method is provided, comprising the following steps performed by a computer: receiving a plurality of facial images with at least partially different brightness levels; determining a parameter value for each of the plurality of facial images for a plurality of image parameters; assigning each of the plurality of facial images to one of a plurality of person groups if, for each of the plurality of image parameters, its parameter value lies within a parameter value range of the respective image parameter specific to the person group; selecting the facial images in a person group with the largest number of assigned facial images; and selecting, from the facial images in the selected person group, the facial image which has a highest parameter value of a selected image parameter of the plurality of image parameters as the optimal facial image.

[0011] According to the method according to the invention, a plurality of facial images of a person's face are received with at least partially different brightness levels. A facial image is an image of the person's face, for example, a portrait. The brightness levels of the facial images can be varied, for example, by changing the illumination intensity when capturing at least two or more images or, if sufficient information depth is available, by generating a plurality of individual images with different brightness levels from a facial image. Due to the different brightness levels, the image quality, such as the contrast or brightness of the face, and in particular the representation of skin tones, varies in each facial image.

[0012] For each facial image, several image parameters are determined. Each image parameter is assigned a parameter value. These image parameters can include, for example, entropy, sharpness, and brightness. However, the parameters are not limited to the parameters mentioned above, and other relevant parameters may exist.

[0013] The inventors have now discovered that image parameters for specific groups of people correlate, so that if the parameter values ​​for several parameters of a facial image lie within certain parameter value ranges, the images can be assigned to a group of people, with the facial images within the group meeting certain quality requirements. For example, if each parameter for a facial image has a value that lies above a lower limit for the corresponding parameter value, the image can be assigned to a group of people. The facial image then has a predetermined image quality, such as certain contrast values ​​and / or certain brightness values ​​of the face.

[0014] A group of people can, for example, comprise an ethnic group. Ethnic groups can be characterized, for example, by the skin tones of the people within the group. Different ethnic groups can have different light or dark skin tones and / or different colored skin tones. Within an ethnic group, the skin tones are at least similar. Furthermore, a group of people can be dependent on the gender or age of the people. Even within these groups, the skin tones of the people are comparable.

[0015] The parameter value ranges typical for a group of people can be determined in advance by reference measurements on a large number of test subjects.

[0016] The facial images assigned to the person group with the largest number of assigned facial images are selected. In other words, the facial images are filtered according to the person group with the most frequent facial images. This narrows down the facial images to a single person group. Furthermore, the images assigned to the person group that most likely corresponds to the person's real group are selected. Images that were assigned to groups other than the person's real group due to varying brightness levels are thus eliminated. This makes the method more reliable. From the facial images assigned to the person group with the largest number of facial images, the facial image with the highest parameter value of a selected parameter, such as entropy, is selected.

[0017] Thus, from the multiple facial images with different brightness levels, the facial image with the optimal brightness level is selected, i.e., the contrast and / or brightness, particularly of the skin tones, that meet the specifications. This results in a facial image with improved image quality, particularly with improved representation of the skin tones. Thus, the initially stated problem is solved.

[0018] Further advantageous aspects and embodiments of the invention are described below.

[0019] According to one aspect, the method comprises the following steps: forming, for each of the selected facial images in the person group with the largest number of associated facial images, a sum of the parameter values ​​of the plurality of image parameters, and wherein selecting the facial image from the facial images in the person group with the largest number of associated facial images as the optimal facial image comprises: selecting the facial image whose parameter value of the selected image parameter has the highest value and whose sum of the parameter values ​​has the highest value.

[0020] At least some of the facial images within a group of people may have the same or at least similar parameter values ​​for the selected image parameter. For example, multiple images may have the same brightness level. By selecting the facial image with the highest parameter value for the selected image parameter and, in addition, the sum of the parameter values ​​has the highest value, the facial image of the group of people with the best image quality is selected, even if multiple facial images with the highest parameter value for the selected image parameter exist within the group of people. Furthermore, this provides a selection criterion in case multiple facial images within the group of people with the most facial images have the same highest parameter value for the selected parameter.

[0021] According to one aspect, forming the sum comprises: normalizing, for each of the selected facial images in the person group with the largest number of associated facial images and for each image parameter, the parameter value to the maximum parameter value of one of the plurality of selected facial images; forming the sum of the parameter values ​​by summing the normalized parameter values; and normalizing the summed parameter values ​​to a number of the image parameters.

[0022] This means that the sum of the parameter values ​​is weighted to the parameter with the greatest influence on image quality.

[0023] According to one aspect, if for each of the plurality of facial images the parameter value of at least one image parameter lies outside the parameter value range for each of the person groups, the method comprises: determining an average value from the parameter values ​​of the selected image parameter for the plurality of facial images; selecting the facial image whose parameter value of the selected image parameter has the smallest difference from the average value as the optimal facial image.

[0024] In this case, none of the facial images can be assigned to a group of people because at least one parameter value in each case lies outside the parameter value range. By selecting the facial image whose parameter value of the selected image parameter has the smallest difference from the mean value as the optimal facial image, the facial image with optimal image quality can still be selected from the multiple facial images with at least partially different brightness levels.

[0025] According to one aspect, the selected image parameter is entropy. Entropy is a measure of the distribution of brightness values, in particular gray values, within the image histogram. In other words, the entropy of a facial image is a measure of the information content of the facial image. High entropy indicates a high information content in the facial image; low entropy indicates a low information content in the facial image.

[0026] Thus, by selecting the facial image with the highest entropy value as the selected parameter, the facial image with the highest information content is selected, which is an important indicator of the facial image quality. Thus, an overall facial image with improved image quality is achieved.

[0027] In some aspects, the group of people is an ethnic group and / or a gender group and / or an age group. The inventors have discovered that, especially for these groups of people, the image parameter values ​​are particularly strongly correlated. For example, for the ethnic group, the parameter value ranges for the multiple parameters exhibit a particularly strong correlation depending on the person's ethnicity, e.g., whether the person has a darker or lighter skin color. The same applies to gender and age groups.

[0028] In one aspect, the plurality of image parameters include a brightness, an entropy, and a sharpness.

[0029] The inventors discovered that these image parameters correlate particularly strongly for different groups of people. Furthermore, these image parameters have a relatively large influence on image quality. These image parameters can therefore be used to assign a facial image to a group of people particularly well.

[0030] The brightness is determined by the number of brightness values ​​in the histogram of the facial image or at least parts of it.

[0031] The sharpness of an image is determined by the spatially dependent contrast range at contrast transitions in the facial image, or at least parts of it. The greater the number of contrast differentiation levels at the transitions, the lower the sharpness. Conversely, the lower the number of contrast differentiation levels, the greater the sharpness. The greatest possible sharpness is achieved by direct black / white transitions.

[0032] In one aspect, the parameter range for the brightness is formed by a lower and an upper brightness limit value, wherein the parameter range for the entropy is formed by a lower and an upper entropy limit value, and wherein the parameter range for the sharpness is limited only by a lower image sharpness limit value.

[0033] In other words, the parameter value ranges for entropy and image brightness have a lower and upper limit. The parameter range for sharpness can only have a lower limit. Thus, a facial image can have at least sufficient sharpness for assignment to a group of people. An upper limit for sharpness is formed by the definition of sharpness and the other parameters with values ​​in the corresponding parameter ranges. For parameter values ​​of entropy and brightness within the corresponding parameter ranges, high sharpness is advantageous for image quality. Since at least entropy correlates with sharpness, the upper limit value of sharpness for assignment to a group of people is at least partially limited by the parameter value range of entropy.

[0034] In some aspects, the sharpness parameter range may be limited by a lower and upper limit value. In some aspects, the parameter value range of each parameter is limited by a lower and upper limit value.

[0035] In one aspect, the method comprises determining for each of the plurality of facial images a skin-facial region, wherein the determining of the parameter values ​​is performed in the skin-facial region.

[0036] Thus, the areas of the face with visible skin are taken into account when determining the parameter values. A skin-facial region is a region of the face with skin as its surface. Thus, the facial image whose skin tone reproduction meets the specified requirements is selected as the optimal facial image.

[0037] In one aspect, the method further comprises generating the plurality of facial images having different brightness levels by capturing the plurality of facial images of the person's face using one or more imaging devices and changing an illumination intensity of the face in at least two, preferably each, of the plurality of facial images using an illumination device.

[0038] By varying the illumination intensity of at least two, preferably each, of the multiple facial images during recording, the multiple facial images can be generated with varying brightness levels while maintaining particularly high image quality for the respective images. An imaging device can be, for example, a camera. An illumination device can be, for example, a lamp directed at the person's face, which illuminates the face with a predetermined intensity.

[0039] In one aspect, the method may further comprise: generating image data from the face image selected as the optimal face image.

[0040] Such a method can provide a method for determining image data representing a digital biometric passport image for a security document.

[0041] In some aspects, the method further comprises transmitting the image data to a personalization device configured to personalize a security document, such as an ID card. Such a personalization device may, for example, be configured with a printing device by means of which the selected facial image is printed on the security document. Such a personalization device may be a device configured to store the selected facial image in a memory, in particular a storage device of the security document.

[0042] In some aspects, the method may include: printing, by the personalization device, the color image onto a security document based on the image data.

[0043] Alternatively or additionally, the method may comprise: storing the image data by the personalization device in a memory, in particular a storage device of the security document. This personalizes the security document and enables identification of the person based on the image data or the selected facial image.

[0044] Such a method and such a personalization device is exemplified in the European application EP 4 099 281 A1 , the content of which regarding the personalization device and / or the method for printing a security document and / or the method for determining passport photo data is incorporated herein.

[0045] Furthermore, the object mentioned at the outset is achieved by a device, in particular a computer, which is set up and designed to carry out the method described above, as well as a computer program product comprising instructions which, when executed by a data processing device, in particular the device, cause the latter to carry out the method steps of the method described above.

[0046] Furthermore, the object mentioned at the outset is achieved by a system comprising: a device that is configured and designed to carry out the method described above; one or more imaging devices that are configured to record the plurality of images of the person's face; and one or more illumination devices that are configured to change an illumination intensity of the face in at least two, preferably each, of the plurality of images. Such a system enables a person to independently record multiple facial images with different brightness levels. The one or more imaging devices and the one or more illumination devices can be coupled to the device wirelessly or wired.The device may be designed and configured to control the one or more imaging devices and the one or more illumination devices such that the plurality of facial images are recorded at least partially at different brightness levels.

[0047] Such a system can also be referred to as a self-enrollment system or facial image capture device. This system is specifically designed to allow a person to independently capture a facial image without the presence of an authorized person and to enroll. Furthermore, such a system can be the aforementioned personalization device.

[0048] Thus, a facial image can be taken independently by a person of their face and a facial image can be selected without another authorized person being present.

[0049] Description of implementation examples

[0050] In the following, further properties, features and advantages of the invention will become clear by describing preferred embodiments of the invention with reference to the accompanying exemplary drawings.

[0051] Fig. 1 shows an example of a schematic block diagram of the steps of a method;

[0052] Fig. 2 is a schematic representation of an arrangement for a device for determining an image data set for facial images; and

[0053] Fig. 3 is a schematic representation of a further arrangement for a device for determining an image data set for facial images.

[0054] Fig. 1 shows an example of a schematic block diagram of a method for determining a facial image with improved image quality.

[0055] In a step S1, several facial images of a person's face are captured by an imaging device such as a camera, in particular a video camera, at at least partially different brightness levels. For this purpose, the illumination intensity can be adjusted for several or each facial image using an illumination device.

[0056] In a step S2, a plurality of image parameters are determined for each of the multiple facial images in the corresponding facial image. For this purpose, a corresponding parameter value is determined for each of the image parameters. The image parameters are preferably determined in the skin-facial regions of the face in the respective image. The plurality of image parameters can include an entropy, a sharpness, and a brightness in the respective facial image, in particular its skin-facial regions.

[0057] In a step S3, the multiple facial images are assigned to a group of people based on the parameter values ​​of the multiple parameters. A group of people can, for example, include an ethnic group, an age-specific group, and / or a gender-specific group.

[0058] Assigning facial images to groups of people involves comparing the parameter values ​​for each facial image with a respective parameter value range for a group of people.

[0059] If the parameter value of each parameter of a facial image lies within the parameter value range of a person group, the facial image can be assigned to that person group. For example, the parameter value ranges for entropy and brightness can be bounded by a lower and an upper limit, while the parameter value range for sharpness is bounded by only a lower limit.

[0060] In step S4, the facial images assigned to the person groups are filtered for the person group with the largest number of assigned images. For this purpose, the facial images in the person group with the largest number of facial images are selected.

[0061] In step S5, a sum of the parameter values ​​is calculated for each of the selected facial images. The sum is calculated by Sum = 1 / N*Sum(Pi / Pmax), where N is the number of parameters, Pi is the parameter value for the i-th parameter, and Pmax is the maximum parameter value for the respective parameter within the selected facial images. In other words, the sum for each facial image is calculated by summing the parameter values ​​normalized to the maximum respective parameter value of the facial images and normalizing the sum to the number of parameters whose parameter values ​​were summed.In a step S6, the facial images are sorted according to the level of a parameter value of a selected parameter, wherein if the parameter value for the selected parameter of at least two facial images is the same, these facial images with the same selected parameter value of the selected parameter are further sorted according to the level of the sum determined in step S5.

[0062] In step S7, the facial image with the highest parameter value for the selected parameter is selected from the sorted facial images. If multiple facial images have the same highest parameter value for the selected parameter, the facial image with the highest parameter value for the selected parameter and with the highest sum value is selected.

[0063] The facial image selected in this way is found to be the facial image with optimal image quality.

[0064] In a step S8, if none of the facial images can be clearly assigned to a group of people, ie if for each of the facial images at least one parameter value lies outside the corresponding parameter value range for each of the groups of people, the mean value of a selected illumination parameter, e.g. the entropy, is determined from the facial images and a difference to the mean value is determined for the parameter value of the selected parameter for each facial image.

[0065] The facial image whose parameter value of the selected parameter shows the smallest difference from the mean value is selected as the facial image with optimal image quality.

[0066] Fig. 2 shows a schematic representation of an arrangement for a device for determining an image data set for a facial image. With the aid of an imaging device 1, which, for example, is formed with a camera, in particular a video camera, and in the example shown is provided as part of a facial image recording device 2, facial images, such as portrait images, are recorded for a person 3 as part of a video image recording. With the aid of a data processing device 4, such as a computer, which is connected to the facial image recording device 2, the facial images contained in the video data stream are stored at a refresh rate in a storage device 5 of the facial image recording device 2 as digital facial images.

[0067] Subsequently, the apparently inappropriate method is carried out in the data processing device 4 in order to generate an image data set from the sequence of facial images.

[0068] In the embodiment of Fig. 2, the facial image recording device 2 is connected to a personalization device 6, to which the image data set can be selectively transmitted. The personalization device 6 is configured to personalize a security document 7. For this purpose, the personalization device 6 can be configured with a printing device by means of which the biometric passport photo is printed on the security document 7. Alternatively or additionally, the passport photo data can be stored in a storage device 8 of the security document 7. This personalized the security document and enables personal identification based on the biometric passport photo.

[0069] If problems are identified for the method, for example because a planned test was negative, in particular because the facial images do not allow parameters to be determined, control data can then be generated in the facial image recording device 2 in order to influence the further procedure such that the method is aborted, repeated, or extended. In one embodiment, it can be provided that the control data are configured to control an output via an output device 9 to the person 3 such that a user instruction for correcting the error is issued to the person 3, which can include a visual and / or acoustic instruction to the person, for example a request to assume a different pose, to remove a headgear, and / or a request to open both eyes.The output device 9, which can be formed, for example, with a display, can optionally be integrated into the facial image recording device 2, for example when formed in a terminal device for capturing personal data for the security document 7. The output device 9 can be designed as an input / output device for receiving user inputs.

[0070] Alternatively or additionally, in response to the detection of an error, control data can be generated which are configured to improve an operating parameter of the facial image recording device 2 and / or of an illumination device associated therewith for further facial images within the scope of the video image recording, for example for improved illumination of the portrait images.

[0071] Fig. 3 shows an alternative embodiment of the arrangement of Fig. 2, in which a connection to the internet 10 is provided instead of a connection to a personalization device 2. The image data set can be transmitted to a server 11 for further use via the internet connection. For example, the server 11 can be part of a network of a service provider that uses the image data set for the biometric identification of the person as an authorized user of a service. Corresponding methods for using an image data set indicating a biometric passport photograph are known in principle to those skilled in the art in various embodiments.

[0072] In particular in this context, but also in other embodiments, the facial image recording device 2 can be formed with a personal electronic device of the person, for example a computer, a mobile phone, a tablet computer or the like, on which a corresponding software application (app) runs to carry out the method.

[0073] The features disclosed in the above description, the claims and the drawings may be important for the realization of the various embodiments both individually and in any combination.

[0074] List of reference symbols:

[0075] S1 - S8 process steps

[0076] 1 imaging device

[0077] 2 Facial image recording device 3 Person

[0078] 4 Data processing facility

[0079] 5 Storage device

[0080] 6 Personalization setup

[0081] 7 Security document 8 Storage device

[0082] 9 Output device

[0083] 10 Internet

[0084] 11 servers

Claims

Claims 1. A method comprising the following steps performed by a computer: - Receiving multiple facial images with at least partially different brightness levels; - determining a parameter value for each of the plurality of facial images for a plurality of image parameters; - Assigning each of the plurality of facial images to one of a plurality of groups of persons if, for each of the plurality of image parameters, its parameter value lies within a parameter value range of the respective image parameter specific to the group of persons; - selecting the facial images in a group of people with the largest number of associated facial images; and - selecting from the facial images in the selected person group of the facial image as an optimal facial image which has a highest parameter value of a selected image parameter of the plurality of image parameters.

2. The method according to claim 1, wherein the method comprises: - forming, for each of the selected facial images in the person group with the largest number of associated facial images, a sum of the parameter values of the plurality of image parameters, and wherein selecting the facial image from the facial images in the person group with the largest number of associated facial images as the optimal facial image comprises: - Select the facial image whose parameter value of the selected image parameter has the highest value and whose sum of the parameter values has the highest value.

3. The method of claim 2, wherein forming the sum comprises: - Normalising, for each of the selected face images in the person group with the largest number of associated face images and for each image parameter, the parameter value to the maximum parameter value of one of the plurality of selected face images; and - Forming the sum of the parameter values by summing the normalized parameter values and normalizing the summed parameter values to a number of image parameters.

4. The method according to any one of the preceding claims, wherein, if for each of the plurality of facial images the parameter value of at least one image parameter is outside the parameter value range for each of the person groups, the method further comprises: - determining an average of the parameter values of the selected image parameter for the plurality of facial images; - Selecting the facial image as the optimal facial image whose parameter value of the selected image parameter has the smallest difference from the mean value.

5. Method according to one of the preceding claims, wherein the selected image parameter is an entropy.

6. Method according to one of the preceding claims, wherein the group of people is an ethnic group and / or a gender group and / or an age group.

7. The method according to any one of the preceding claims, wherein the plurality of image parameters comprise a brightness, an entropy and a sharpness.

8. The method according to claim 7, wherein the parameter range for the brightness is formed by a lower and an upper brightness limit value, wherein the parameter range for the entropy is formed by a lower and an upper entropy limit value and wherein the parameter range for the sharpness is limited only by a lower image sharpness limit value.

9. The method of any preceding claim, further comprising: determining for each of the plurality of facial images a skin-facial region, and wherein the determining of the parameter values is performed in the skin-facial region.

10. The method according to any one of the preceding claims, further comprising: generating the plurality of facial images with different brightness levels by capturing the plurality of facial images of the person's face using one or more imaging devices and changing an illumination intensity of the face in at least two, preferably each, of the plurality of facial images using an illumination device.

11. Device, in particular a computer, wherein the device is arranged and designed to carry out the method according to one of claims 1-9.

12. Computer program product comprising instructions which, when executed by a data processing device, in particular the device according to claim 11, cause the device to carry out the method steps according to at least one of claims 1 to 9.

13. System comprising: - a device according to claim 11; - one or more imaging devices configured to capture the plurality of images of the person's face; and - one or more illumination devices configured to change an illumination intensity of the face in at least two, preferably each, of the plurality of images.

Images