Fraudulent image detector and computer-implemented method of detecting a fraudulent image

Abstract

A fraudulent image detector (4) comprising: a first segmenter (6) configured to calculate first probability data from an image showing personal identifiable information of an individual, the first probability data indicating, for each pixel of the image, a probability that the pixel shows a security pattern; a second segmenter (8) configured to calculate second probability data from the image, the second probability data indicating, for each pixel of the image, a probability that the pixel is part of a foreground region showing the personal identifiable information or is part of a background region showing no personal identifiable information;a classifier (12) configured to calculate score data from the first probability data and the second probability data, the score data indicating a probability that the background region as a whole shows a security pattern and that the foreground region as a whole shows no security pattern; and an output module (14) configured to output a result indicating that the image is fraudulent whenever the probability indicated by the score data is greater than a threshold.;

Description

Title of the invention: Fraudulent image detector and computer-implemented method for detecting a fraudulent image Technical field

[0001] The present invention relates to a fraudulent image detector and a computer-implemented method of detecting a fraudulent image. CONTEXT

[0002] One of the most significant frauds in identity documents is related to portrait photo swapping, which replaces genuine facial Personally Identifiable Information (PII) with a fraudulent face photo. Photo swaps can consist of either digital or physical substitution of the genuine portrait photo. Such substitution leaves traces that can be very subtle and invisible to human verifiers / operators.

[0003] Known solutions for detecting such fraud involve analysis of chromaticity statistics at the pixel level and follow a variation in hue over video frames. Such solutions have drawbacks, including the requirement for multiple frames. SUMMARY

[0004] An objective of the present invention is to provide a solution for detecting fraud in a single image.

[0005] There is therefore provided a fraudulent image detector comprising: a first segmenter configured to calculate first probability data from an image showing personal identifiable information of an individual, the first probability data indicating, for each pixel of the image, a probability that the pixel shows a security pattern; a second segmenter configured to calculate second probability data from the image, the second probability data indicating, for each pixel of the image, a probability that the pixel is part of a foreground region showing the personal identifiable information or is part of a background region showing no personal identifiable information;a classifier configured to calculate score data from the first probability data and the second probability data, the score data indicating a probability that the background region as a whole shows a security pattern and that the foreground region as a whole shows no security pattern; and an output module configured to output a result indicating that the image is fraudulent whenever the probability indicated by the score data is greater than a threshold. ;

[0006] The fraudulent image detector may further include optional features, which are listed below and which may be taken individually or in combination whenever it makes sense.

[0007] Optionally, the first probability data comprises, for each pixel of the image: a probability that the pixel shows a hologram, and a probability that the pixel shows an overprint, the overprint preferably consisting of points, lines or a combination thereof.

[0008] Optionally, the score data includes: hologram score data indicating a probability that the background region shows a hologram and the foreground region shows no hologram, overprint score data indicating a probability that the background region shows an overprint and the foreground region shows no overprint, and the result indicates that the image is fraudulent whenever the probabilities indicated by the hologram score data or the overprint score data are greater than respective thresholds.

[0009] Optionally, the image originates from an identity document issued by an authority, and the output module is further configured to: calculate a mask from the first probability data, the mask estimating the location of a security pattern in the image, compare the mask and authority data indicating the location of a reference security pattern in identity documents issued by the authority, and output the result indicating that the image is fraudulent whenever the estimated location and the reference location disagree.

[0010] Optionally, the calculation of the mask includes a thresholding of the probability data, such that the mask indicates, for each pixel of the image, whether the pixel shows a security pattern or not.

[0011] Optionally, at least one of the first segmenter, the second segmenter and the classifier is a convolutional neural network.

[0012] Optionally, the personally identifiable information includes a biometric characteristic, such as the individual's face or a fingerprint of the individual.

[0013] A second subject matter of the present invention is a computer-implemented method of detecting a fraudulent image, the method comprising the steps of: calculating probability data from an image showing personally identifiable information of an individual, the probability data indicating, for each pixel of the image, a probability that the pixel shows a security pattern; calculating second probability data from the image, the second probability data indicating, for each pixel of the image, a probability that the pixel is part of a foreground region showing the information personally identifiable information or is part of a background region showing no personally identifiable information; calculating score data from the probability data and the segmentation data, the score data indicating a probability that the background region as a whole shows a security pattern and that the foreground region as a whole shows no security pattern; and outputting a result indicating that the image is fraudulent whenever the probability indicated by the score data is greater than a threshold.

[0014] A third object of the present invention is a computer program product comprising program code instructions for carrying out the method constituting the second object, when the program is executed by a computer.

[0015] A fourth object of the present invention is a non-transitory computer-readable medium comprising program instructions for causing a computer to carry out the method constituting the second object. DESCRIPTION OF DRAWINGS

[0016] Other characteristics and advantages of embodiments will be detailed in the following description, which is purely illustrative and not exhaustive, and which must be read in conjunction with the attached drawings, comprising the following figures.

[0017] [Fig.l] is an image of an identity document according to one embodiment.

[0018] [Fig.2] schematically shows a system according to one embodiment.

[0019] [Fig.3] shows steps of a method according to one embodiment.

[0020] [Fig.4] comprises an image showing verifiable personal information, and a mask calculated from said image by the system of [Fig.2].

[0021] DETAILED DESCRIPTION OF AT LEAST ONE EMBODIMENT Identity document

[0022] An identity document D intended to prove the identity of an individual is shown in [Fig.l]. The identity document D contains personally identifiable information.

[0023] The identity document D was originally issued by an authority to an individual, all personally identifiable information in the identity document D normally relates to that individual, who is the legitimate holder of the identity document 1.

[0024] The identity document D includes a photograph P, called a primary portrait photograph, showing the face of an individual. The identity document D also includes text (name, date of birth, nationality, etc.). The face and the text are part of the personally identifiable information.

[0025] [Fig. 1] is a raw image of identity document D. This raw image includes several regions: • a foreground region F showing the individual's face (this region therefore shows part of the photograph). • a background region B showing another part of the photograph which does not carry any personally identifiable information; in the example shown in [Fig.l], this background region surrounds the foreground region. • a background region T that is outside the photograph P. Text may appear in the background region T. • a border region E showing a border between the photograph P and the background region T. The border region E is generally rectangular. The border region E may include an inner border region showing an inner border surrounding the photograph P and an outer border region showing an outer border surrounding the inner border.

[0026] The identity document D includes at least one security reason. The or each security reason depends on the authority that issued the identity document D. In other words, all identity documents of the same type issued by the same authority include security reasons that are similar or even identical.

[0027] The or each security pattern is superimposed over the personally identifiable information on the identity document. Therefore, some pixels in the foreground region F show personally identifiable information (e.g., the face), and some other pixels in the foreground region F show a security pattern overlying the personally identifiable information (e.g., the face), and in such a case, the pixel is considered to comprise both as a weighted sum.

[0028] In other regions, such as background region B, some pixels may show a security pattern and some other pixels may not.

[0029] In the following, two different types of security patterns will be discussed: holograms and overlays. As those skilled in the art know, a hologram is a view-dependent pattern. An overlay may consist of lines, dots, or a combination of lines and dots.

[0030] The identity document may include a hologram or an overprint, or a combination of the two security features. System

[0031] Referring to [Fig.2], a system 1 comprises a camera 2, a pre-processing module 3 and a fraudulent image detector 4.

[0032] The camera 2 is configured to acquire an image showing personally identifiable information of an individual. The personally identifiable information may be or include the face of the individual.

[0033] More specifically, the camera 2 can be used to acquire an image of an identity document, such as the identity document D shown in [Fig.l]. The identity document D is a driving license, but the camera 2 can be used to acquire an image of an identity document of another type, such as an identity card, a passport or a driving license.

[0034] The preprocessing module 3 is configured to preprocess a raw image acquired by the camera 2, thereby producing an image of interest to be evaluated by the fraudulent image detector 4. As detailed below, the image produced by the preprocessing module may be a part of interest of the raw image. In any case, this image shows personal identifiable information of an individual.

[0035] A function performed by the fraudulent image detector is to detect whether an image showing verifiable personal information is fraudulent or not. The fraudulent image detector can either work on an image of interest delivered by the preprocessing module 3, or work on a raw image acquired by the camera 2. In other words, the preprocessing module 3 is optional.

[0036] The fraudulent image detector 4 comprises a first segmenter 6, a second segmenter 8 and a fraud evaluation module 10.

[0037] The first segmenter 6 is configured to calculate first probability data from the image, the first probability data indicating, for each pixel of the image, a probability that the pixel shows a security pattern. The probability data will be detailed below.

[0038] The first segmenter 6 is a first convolutional neural network.

[0039] The first convolutional neural network may be trained by supervised learning using first training data. The first training data comprises a set of first training images showing personally identifiable information, and which may or may not show at least one security pattern. Each first training image is annotated with labels used as ground truth. The labels indicate, for each pixel of the image, whether the pixel shows a security pattern or not. A label associated with a pixel may take, for example, three possible values: • a first value indicating that the pixel shows a hologram, • a second value indicating that the pixel shows overprinting, • a third value indicating that the pixel shows no pattern of security.

[0040] The first value and the second value can be equal (in this case, the label will indicate that a pixel shows a hologram or an overprint, without distinguishing the two cases).

[0041] The second segmenter 8 is configured to calculate second probability data from the image, the segmentation data identifying, for each pixel of the image: a foreground region showing the personally identifiable information, and a background region showing no personally identifiable information.

[0042] The second segmenter 8 is a second convolutional neural network distinct and independent from the first convolutional neural network.

[0043] The second convolutional neural network may be trained by supervised learning using second training data. The second training data comprises a set of second training images showing personally identifiable information, and which may or may not show at least one security pattern. Each second training image is annotated with labels used as ground truth. The labels indicate, for each pixel of the second training image, whether the pixel shows personally identifiable information or not. More specifically, a label may indicate a region to which the pixel belongs. A label associated with a pixel may take, for example, five possible values: • a first value indicating that the pixel is part of the background region T, • a second value indicating that the pixel is part of the outer border region, • a third value indicating that the pixel is part of the foreground region F, • a fourth value indicating that the pixel is part of the background region B, • a fifth value indicating that the pixel is part of the inner border region.

[0044] The first training data and the second training data may have images in common. Alternatively, the first training data and the second training data are entirely different. For example, synthetic overlays or holograms may be selectively added into images that are part of the first training data, and not into images of the second training data.

[0045] The first convolutional neural network and the second convolutional neural network may operate in parallel or successively.

[0046] The fraud evaluation module 10 comprises a classifier 12 and an output module 14.

[0047] The classifier 12 is configured to calculate score data from the first probability data (produced by the first segmenter 6) and the second probability data (produced by the second segmenter 8). The score data indicates a probability that the background region B as a whole shows a security pattern and that the foreground region F as a whole shows no security pattern. Unlike the first probability data and the second probability data, which provide information at the pixel level, the score data provides information at a more global level: either at the level of the region of interest or at the level of the document represented in the image.

[0048] Classifier 12 is a third convolutional neural network distinct from the first convolutional neural network and the second convolutional neural network.

[0049] The third convolutional neural network may be trained by supervised learning using third training data. The third training data comprises first and second probability data output by the first segmenter and by the second segmenter in association with labels used as ground truth. The labels indicate whether the first and second probability data come from an image satisfying the following requirements or not: • a background region of the image showing a security pattern and / or no personally identifiable information, and • a foreground region of the image showing personally identifiable information and no security grounds.

[0050] The labels may also indicate whether the first and second probability data come from an image meeting the following requirements or not: • a foreground region of the image showing a security pattern and / or no personally identifiable information, and • a background region of the image showing personally identifiable information and no security grounds.

[0051] The output module 14 is configured to deliver a result indicating that the image introduced into the fraudulent image detector 4 is fraudulent whenever the probability indicated by the score data is greater than a threshold.

[0052] The preprocessing module 3, the first segmenter 6, the second segmenter 8, the classifier 12 and the output module may be separate hardware components, each hardware component comprising a memory storing a computer program and a processor for executing code instructions of the computer program. Alternatively, the modules 3, 6, 8, 12 are parts of a computer program executed by a processor of system 1 and stored in a memory of system 1. Process

[0053] A computer-implemented method using the system 1 described above comprises the following steps.

[0054] In an acquisition step S2, the camera 2 acquires a raw image of an identity document, as described above. It will be assumed in the following that the raw image is the image of [Fig.l], showing the identity document D. The raw image may for example present a standardized card rendering at 400 dpi.

[0055] In a preprocessing step S3, the preprocessing module 3 produces an image of interest from the raw image, the image of interest showing personal identifiable information of an individual.

[0056] The preprocessing step S3 may comprise cropping the image, such that the image of interest is a part of interest of the raw image. The image of interest comprises the photograph P of the individual. The size of the image of interest may be normalized for example to 320x320 after cropping.

[0057] Preferably, the image covers not only the photograph P but also at least part of the background region T. This makes it possible to check the continuity at the border of the photo and to compensate for errors in detecting a portrait (not all portrait photos show a clear rectangular border). A skilled person can, for example, apply the verification / compensation method taught in US 18 / 062,963.

[0058] In a first segmentation step S6, the first segmenter 6 calculates first probability data from the image. The first probability data indicates, for each pixel of the image, a probability that the pixel shows a security pattern.

[0059] The first probability data takes various forms. In one embodiment, the probability data comprises at least one probability map (sometimes referred to as a “heat map” in the literature), each probability map being associated with a respective type of security pattern.

[0060] Each probability map includes, for each pixel in the image, a probability that the pixel exhibits a security pattern of the type associated with the map. In other words, each probability map includes as many probabilities as the number of pixels in the image.

[0061] In one embodiment, the probability data comprises a hologram probability map and an overlay probability map. The hologram probability map comprises, for each pixel in the image, a probability that the pixel exhibits a hologram, the first segmenter being trained and designed to detect holograms, tolerating partial invisibility of holograms. The overprint probability map includes, for each pixel in the image, a probability that the pixel shows an overprint. A pixel is considered to show a single piece of information, not a multiplicity of them. Therefore, all probabilities associated with the same pixels are less than or equal to 1.

[0062] In another embodiment, the probability data comprises a single probability map comprising, for each pixel in the image, a probability that the pixel exhibits a security pattern of any type. In other words, overprints and holograms are not distinguished in this embodiment.

[0063] In a second segmentation step S8, the second segmenter 8 calculates second probability data from the image. The second probability data indicates, for each pixel of the image, a probability that the pixel is part of the foreground region F showing personal identifiable information or is part of the background region B showing no personal identifiable information.

[0064] In one embodiment, the second probability data comprises a single probability map comprising a plurality of probabilities respectively associated with the pixels of the image. Each probability of the map may be a probability that the pixel is part of the foreground region F showing personally identifiable information or a probability that the pixel is part of the background region B showing no personally identifiable information.

[0065] In another embodiment, the second probability data provides fine-grained information. To do this, the second probability data may comprise several probability maps, such as: • a foreground probability map indicating, for each pixel in the image, a probability that the pixel is part of the foreground region F showing the face of an individual, • a background probability map indicating, for each pixel in the image, a probability that the pixel is part of the background region B included in the photograph P on the identity document D but not showing the face of the individual, • an edge probability map indicating, for each pixel in the image, a probability that the pixel is part of the edge region E. • a background probability map indicating, for each pixel in the image, a probability that the pixel is part of the background region T showing part of the identity document D outside the photograph P.

[0066] The first segmentation step S6 and the second segmentation step S8 can be carried out in parallel or successively, in any order.

[0067] In a classification step S12, the classifier 12 calculates score data from the first probability data and the second probability data. Unlike the first probability data and the second probability data, which provide pixel-level information, the score data provides more global information. More specifically, the score data indicates a probability that a particular event of interest occurs in the image. The event of interest is that the background region shows a security pattern and the foreground region shows no security pattern.

[0068] This event of interest reveals an inconsistency that is most likely due to a physical modification of the personally identifiable information. For example, an attacker may have pasted a second photograph over the photograph originally present in the identity document D issued by an authority, the second photograph showing the face of another individual and not showing any security pattern. Alternatively, an attacker may have modified the foreground region F of the image to replace the individual's face with the face of another individual.

[0069] It should be noted that the probability of this event of interest can be indicated in different ways, as illustrated below.

[0070] In a first embodiment, the score data consists of a single score, said score indicating the probability of the event of interest.

[0071] In a second embodiment, the score data comprises two scores: one score indicating a probability that the background region shows a security pattern and another score indicating a probability that the foreground region shows no security pattern.

[0072] In a third embodiment, the score data includes hologram score data indicating a probability that the background region shows a hologram and the foreground region shows no hologram, and overprint score data indicating a probability that the background region shows an overprint and the foreground region shows no overprint.

[0073] The hologram score data may consist of a single hologram score indicating that the following event occurs: the background region shows a hologram, and the foreground region shows no hologram. Alternatively, the hologram score data may comprise two scores: one score indicating a probability that the background region shows a hologram and another score indicating a probability that the foreground region shows no hologram.

[0074] Similarly, the overprint score data may consist of a single overprint score indicating a probability that the following event occurs: the background region shows overprint, and the foreground region shows no overprint. Alternatively, the overprint score data may comprise two scores: one score indicating a probability that the background region shows overprint and another score indicating a probability that the foreground region shows no overprint.

[0075] In summary, the score data can take different forms: a single score directly giving the probability of the event of interest described above, or a set of scores giving partial information and which can be combined to indirectly obtain the probability of the event of interest.

[0076] The output module 14 performs a post-processing step S14. The post-processing step S14 comprises comparing the score data and at least one threshold, and delivering a result, the result having a value that depends on the comparison(s).

[0077] The number of thresholds involved in this step depends on the content of the score data. This step may involve as many thresholds as the number of scores included in the score data. For example, if the score data consists of a single score, said score is compared to a single threshold. No other thresholds are involved. Alternatively, if the score data comprises a hologram score and an overlay score, the hologram score may be compared to a first threshold and the overlay score may be compared to a second threshold, either in parallel or successively (in any order). If the score data comprises four scores, four comparisons are made, either in parallel or successively.

[0078] Regardless of the content of the score data and the number of comparisons performed, the results delivered by the output module 14 indicate that the image is fraudulent whenever the probability of the event of interest occurring (i.e. whenever the background region shows a security pattern and the foreground region shows no security pattern) is greater than a main threshold.

[0079] On the other hand, the results delivered by the output module 14 indicate that the image is not fraudulent whenever the probability that the event of interest occurs (i.e. that the background region shows a security pattern and that the foreground region shows no security pattern) is not greater than said main threshold.

[0080] If the score data consists of a single score, then the primary threshold is the threshold compared to said single score. Otherwise, the primary threshold may be determined from all thresholds to which the multiple scores of the score data were compared.

[0081] The output module 14 advantageously includes the details of the sub-steps below, which are optional.

[0082] The output module 14 calculates a mask from the first probability data. The mask estimates a location of a security pattern in the image. More specifically, the mask may estimate the location of a hologram and / or the location of an overprint in the image. In fact, the output module may calculate a multiplicity of masks, each mask being associated with a security pattern of a specific type.

[0083] The calculation of the mask may include a thresholding of the first probability data. In this case, the mask may be a binary mask, i.e. a mask comprising booleans associated respectively with the pixels of the image.

[0084] The output module 14 compares a mask calculated in the location step and authority data indicating the location of a reference security pattern in identity documents issued by the authority that issued the identity document D (said document D being at least partly represented in the image introduced into the fraudulent image detector 4).

[0085] The result delivered by the output module 14 indicates that the image is fraudulent whenever the estimated location and the reference location disagree.

[0086] In other words, when these additional steps are implemented by the output module 14, the output module 14 makes the decision to report fraud when it considers that at least one of the following events of interest occurs in the image: • background region B shows a security pattern and foreground region F shows no security pattern. • a security pattern is found in the image (using the first segmenter) but its location does not match a reference location decided by the competent authority, i.e. the found security pattern is not located where it should be.

[0087] A third event of interest may be that both the background and foreground regions show no security pattern. This event is likely to reveal fraud (the entire rectangular photo may have been replaced). This third event may be evaluated in post-processing step S14 and may involve one or more additional comparisons between the score data and at least one additional threshold.

[0088] These events of interest can be weighted differently depending on their deviation / significant nature.

[0089] The left part of [Fig.4] includes a PII, in the form of an image showing the face of an individual, and two security patterns extending over said face: a circular hologram and an overprint consisting of lines (note that the white area covering the face of the first individual is not normally present in the image; this white area has been added to [Fig.4] solely for confidentiality purposes). The right part of [Fig.4] contains a mask generated by the fraudulent image detector from the first image. It can be seen that a significant part of the circular hologram and the lines of the overprint were correctly detected by the fraudulent image detector 4.

[0090] The mask(s) generated by the output module 14 may also be used for other verifications such as ghost face matching, etc. Other embodiments

[0091] An embodiment has been described in which the personally identifiable information comprises a primary portrait photograph showing an individual's face. However, other embodiments are possible. The photograph could alternatively refer to a phantom portrait photograph. In addition, the personally identifiable information is not limited to faces, and could be or include other biometric information, for example, a fingerprint.

[0092] The preprocessing module 3 is optional. This means that the first segmenter 6 and / or the second segmenter 8 can process the raw image acquired by the camera 2.

Claims

Claims

1. A fraudulent image detector (4) comprising: • a first segmenter (6) configured to calculate first probability data from an image showing personal identifiable information of an individual, the first probability data indicating, for each pixel of the image, a probability that the pixel shows a security pattern; • a second segmenter (8) configured to calculate second probability data from the image, the second probability data indicating, for each pixel of the image, a probability that the pixel is part of a foreground region showing the personal identifiable information or is part of a background region showing no personal identifiable information;• a classifier (12) configured to calculate score data from the first probability data and the second probability data, the score data indicating a probability that the background region as a whole shows a security pattern and that the foreground region as a whole shows no security pattern; and • an output module (14) configured to output a result indicating that the image is fraudulent whenever the probability indicated by the score data is greater than a threshold.;

2. Detector (4) of fraudulent images according to the preceding claim, the first probability data comprising, for each pixel of the image: • a probability that the pixel shows a hologram, and • a probability that the pixel shows an overprint, the overprint preferably consisting of points, lines or a combination thereof.

3. Detector (4) of fraudulent images according to any one of the preceding claims: • the score data comprising: • hologram score data indicating a probability that the background region shows a hologram and the foreground region shows no hologram, • overprint score data indicating a probability that the background region shows an overprint and the foreground region shows no overprint; and • the result indicating that the image is fraudulent whenever the probabilities indicated by the hologram score data or by the overprint score data are greater than respective thresholds.

4. A fraudulent image detector (4) according to any one of the preceding claims, the image originating from an identity document (D) issued by an authority, and the output module (14) being further configured to: • calculate a mask from the first probability data, the mask estimating the location of a security pattern in the image, • compare the mask and authority data indicating the location of a reference security pattern in identity documents issued by the authority, • output the result indicating that the image is fraudulent whenever the estimated location and the reference location disagree.

5. Detector (4) of fraudulent images according to the preceding claim, the calculation of the mask comprising a thresholding of the probability data, such that the mask indicates, for each pixel of the image, whether the pixel shows a security pattern or not.

6. Fraudulent image detector (4) according to any one of the preceding claims, at least one element among the first segmenter (6), the second segmenter (8) and the classifier (12) being a convolutional neural network.

7. A fraudulent image detector (4) according to any preceding claim, the personally identifiable information comprising a biometric characteristic.

8. Detector (4) of fraudulent images according to the preceding claim, the biometric characteristic being at least a part of the face of the individual or a fingerprint of the individual.

9. A computer-implemented method of detecting a fraudulent image, the method comprising the steps of: • calculating (S6) probability data from an image showing personal identifiable information of an individual, the probability data indicating, for each pixel of the image, a probability that the pixel shows a security pattern; • calculating (S8) second probability data from the image, the second probability data indicating, for each pixel of the image, a probability that the pixel is part of a foreground region showing the personal identifiable information or is part of a background region showing no personal identifiable information;• calculating (S 12) score data from the probability data and the segmentation data, the score data indicating a probability that the background region as a whole shows a security pattern and that the foreground region as a whole shows no security pattern; and • outputting (S 14) a result indicating that the image is fraudulent whenever the probability indicated by the score data is greater than a threshold.;

10. A computer program product comprising program code instructions for carrying out the method according to the preceding claim when the program is executed by a computer.

Images