Method, device and system for detecting facial manipulation

Abstract

A method for detecting facial manipulation, comprising the following steps performed by a computer (1): receiving at least one image pair consisting of a color image of a face and a simultaneously acquired thermal image of the face; adjusting the color image and the thermal image so that the face in the color image is congruent with the face in the thermal image; generating a combined image by combining the adjusted color image and the adjusted thermal image; determining, on the combined image, for one or more classes indicating a type of manipulation, a probability value indicating the probability that manipulation has occurred for the corresponding class; and classifying, based on the probability values, whether facial manipulation has occurred.

Description

[0001] The invention relates to a method for detecting facial manipulation, a device for detecting facial manipulation, a system for detecting facial manipulation, and a computer program product comprising commands that cause a suitably designed device to carry out the method for detecting facial manipulation. background

[0002] A person's face is crucial for their identity. Often, a color photograph of the face of a person to be identified is taken for passport or ID photos. A color photograph can be, for example, an image in the RGB color space, the CMYK color space, or a monochrome image. These images should be authentic, meaning they should reflect the person's actual face, in order to reliably identify the person, for example, on personalized ID cards.

[0003] Furthermore, facial recognition methods are known in which a camera captures an image of a person's face in the visible wavelength range. The resulting image provides a representation of the person's face. By analyzing the geometric arrangement of characteristic facial features, such as the eyes, nose, or mouth, and their position, relative distance, and location within the face—that is, an area on the person's frontal head specific to each individual—a person can be uniquely identified.

[0004] A corresponding image of the person's face can, for example, be taken on site by an employee of a security service provider, in which the employee takes a picture of the face of the person to be identified.

[0005] Recently, the trend has been towards capturing a person's face without the involvement of another person, such as a service provider's employee. This can be achieved through a so-called self-enrollment system, in which the person to be identified independently takes a picture of their face without the presence of anyone else. A potential identification or facial recognition process can then be automated by software using this captured image of the person's face.

[0006] Since the image of the face of the person to be identified is now taken without the presence of an authorized person, there is a possibility that the person to be identified may manipulate the system.

[0007] In the worst case, the person to be identified tries to deceive the system by presenting it with a different identity than their own, using a different or at least partially falsified face.

[0008] Such attempts at deception can consist, for example, of the person to be identified bringing an image of another person at least partially into the camera's field of view, putting on a mask, holding a printed or digital image with the image of another or even a non-real person into the camera's field of view, or concealing certain facial features, for example, with sunglasses, an eye patch, or similar items.

[0009] Such deception attempts result in the image captured by the camera showing a different or at least partially distorted face of the person to be identified.

[0010] This creates the risk that the image taken of the face of the person to be identified will show a different or distorted face of the person, thus falsely representing the identity of the person to be identified. Summary

[0011] Against this background, one of the tasks of the invention is to detect facial manipulations when taking a picture.

[0012] The problem is solved by a method for detecting facial manipulation, which comprises the following steps performed by a computer: receiving at least one image pair with a color image of a face and a simultaneously acquired thermal image of the face; adjusting the color image and the thermal image so that the face in the color image is congruent with the face in the thermal image; generating a combined image by combining the adjusted color image and the adjusted thermal image; determining, for one or more classes indicating a type of manipulation, a probability value for each of the combined images, which indicates the probability that manipulation has occurred for the corresponding class; and classifying, based on the probability values, whether facial manipulation has occurred.

[0013] The invention is based on the inventors' finding that significantly greater safety can be achieved by combining a color image in the visible wavelength range with a thermal image in the infrared wavelength range, which indicates a heat distribution. A thermal image could, for example, be a FLIR image.

[0014] Both the color image and the simultaneously recorded thermal image show a similar section, ideally the same section, of the face of a person to be identified.

[0015] The color image and the thermal image are adjusted so that the face in the color image and the face in the thermal image are congruent. Congruent means that two objects are identical, i.e., that the face is identical in both images. In other words, all points of the face in the color image correspond to the points of the face in the thermal image. The face in the color image and the face in the thermal image can be approximately congruent. Congruent also includes minor deviations of the faces in the range of 0.00001%–10%, for example, 0.01, 0.1, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10%, or in other words, in the range of 0.0000001–10 mm, for example, 0.01, 0.1, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 mm.

[0016] The adjusted thermal image and the adjusted color image are combined, for example, by stacking. The combined image contains the face, with the information from the color image linked to the information from the thermal image for each available point of the face in the combined image. Thus, location-dependent information in the visible wavelength range is linked with information in the infrared wavelength range. The combined image can, for example, take the form of a tensor in which the information from the color image and the thermal image is combined. When the color image and the thermal image are combined by stacking, the combined image can also be referred to as a stacked image.

[0017] The combined image makes it easier to determine whether facial manipulation has occurred, since a face usually has a specific heat distribution.

[0018] If a person uses a reproduction of another person's face, such as a print or a digital display on a screen, or covers certain areas of their own face, the thermal image information in the combined image changes because the heat distribution of the captured face is altered compared to the unaltered face due to the facial manipulation.

[0019] These different types of manipulations exhibit a specific heat distribution. Thus, a class with a specific heat distribution can be assigned to each type of manipulation. For example, a first class might include holding a printed image of a face up to the camera. The printed image exhibits a first, location-dependent specific heat distribution. A second class might include putting on a mask. The image of a face with a mask on exhibits a second, location-dependent specific heat distribution. A third class might, for example, include putting on glasses with image elements. The image of a head with such glasses on exhibits a third, location-dependent specific heat distribution.Such location-dependent specific heat distributions can be determined for different types of manipulations, whereby the number of types of manipulations is not limited to the examples mentioned and a multitude of such classes, i.e. types of manipulations, can be formed.

[0020] The combined image thus allows a probability value to be determined for each type of manipulation, indicating the probability that a particular type of manipulation is present.

[0021] The specific probability value for each class, i.e., each type of manipulation, allows for classification of whether manipulation has occurred.

[0022] This solves the aforementioned task, in which it is possible to recognize or classify whether facial manipulation is present.

[0023] Further advantageous aspects and embodiments of the invention are described below.

[0024] According to one aspect, matching the color image and the thermal image involves the following steps: determining characteristic facial features of the face in the color image; scaling dimensions of the color image so that the distance between any two characteristic facial features of the face in the color image has a predetermined value; perspectivally transforming the thermal image using a coordinate transformation so that the perspective of the thermal image and the perspective of the color image coincide; and scaling dimensions of the thermal image to the dimensions of the scaled color image so that the face in the scaled thermal image is congruent with the face in the scaled color image.

[0025] A coordinate transformation can preferably be an affine mapping.

[0026] This aspect allows the color image and the thermal image to be adjusted particularly precisely so that the face in the color image is congruent or essentially congruent with the face in the thermal image.

[0027] Typically, characteristic facial features are more pronounced and identifiable in the visible spectrum. Furthermore, a color image usually has a higher resolution than a thermal image. This allows for more precise identification of facial features in the color image than in the thermal image. The thermal image is therefore adjusted based on the information in the color image to ensure that the face in the thermal image is congruent with the face in the color image. The distance between two of the face's characteristic features can be a Euclidean distance between the two eyes, for example, on the order of 60 or 100 pixels.

[0028] Since the color image is usually captured by a first camera and the thermal image by a second camera, which is spaced apart from the first and therefore has a slightly different perspective, the thermal image is transformed using a perspective coordinate transformation, in particular an affine transformation, so that the perspective of the first camera (i.e., the color image) and the second camera (i.e., the thermal image) coincide. This improves the accuracy of the process and compensates for mechanical inaccuracies between devices.

[0029] Based on the scaling of the color image, the thermal image can be scaled to the same dimensions as the scaled color image. A dimension can, for example, include the horizontal and vertical dimensions of the corresponding image, but also optionally the resolution of the pixels of the corresponding image.

[0030] After adjusting the color image and the thermal image, the face in the color image and the face in the thermal image are essentially congruent, and both images show the same section.

[0031] According to one aspect, the procedure includes: cropping the scaled color image based on the characteristic facial features, so that the face is positioned at a predetermined location and size in the color image; and cropping the scaled thermal image analogously to the cropping of the color image, so that the cropped thermal image and the cropped color image have the same section.

[0032] Based on the characteristic facial features identified in the color image, the scaled color image and the scaled thermal image can be cropped so that the face is positioned at a predetermined location within the respective image. The cropping can, for example, be designed so that the face in the corresponding color image and thermal image conforms to ICAO standards.

[0033] This ensures that other areas of the body, such as the shoulders or torso and the background, are not used to detect facial manipulation, which could distort the analysis. Furthermore, this guarantees a certain degree of uniformity, which facilitates the analysis, particularly the determination of probability values.

[0034] According to one aspect, the characteristic facial features in the color image are determined using artificial intelligence.

[0035] The application of artificial intelligence has proven particularly effective in recognizing characteristic facial features. In particular, it has been shown that artificial intelligence can be trained especially effectively to identify differences between images. One example of an artificial intelligence capable of recognizing specific facial features is the BlazeFace algorithm.

[0036] According to one aspect, the procedure further comprises: normalizing, by the computer, the probability value for each of the one or more classes to a normalized probability value; and wherein classifying whether facial manipulation is present comprises: summing the normalized probability values ​​for several classes, classifying that facial manipulation is present if the sum of the normalized probability values ​​is greater than a first threshold, and classifying that facial manipulation is not present if the sum of the normalized probability values ​​is less than the first threshold or less than a second threshold that is lower than the first threshold.

[0037] By normalizing the probability value for each type of manipulation or class and summing several probability values ​​for different types of manipulation, an overall probability can be determined with which a manipulation is present.

[0038] The overall probability can be used to classify whether facial manipulation is present.

[0039] The normalization of the probability values ​​to normalized probability values ​​can be carried out, for example, using a softmax function.

[0040] According to one aspect, the determination of a probability value for one or more classes indicating a type of manipulation is carried out using a convolutional neural network.

[0041] Such a convolutional neural network (CNN) has proven to be particularly effective and reliable in determining a probability value for each type of manipulation in the combined image, especially based on the thermal information provided by the thermal image.

[0042] Examples of such convolutional neural networks are the so-called Zalando CNN or the so-called MNIST CNN.

[0043] The convolutional neural network can be trained to determine a probability value for each type of manipulation in the form of a class.

[0044] For each type of manipulation, a large number of images are provided, some depicting a manipulation attempt and some not. For example, for a class of mask manipulation attempt—that is, a type of manipulation where the person to be identified wears a mask—a specific number of images are provided in which the person is not wearing a mask, and a specific number in which the person is wearing a mask. For each image, it is indicated whether it represents a manipulation or not. The large number of inputs allows for the formation of specific patterns, from which a probability value can be derived from a combined image, indicating the likelihood of a manipulation. This process is repeated for each class representing a type of manipulation, and the number of classes is unlimited.The convolutional neural network is therefore trained in advance for each class.

[0045] This allows probability values ​​for the different classes indicating a type of manipulation to be determined from such a combined image.

[0046] Normalizing these probability values ​​has proven particularly useful in conjunction with the probability values ​​output by the convolutional neural network, since the latter typically outputs unnormalized probability values. This results in better comparability and classification.

[0047] According to one aspect, the procedure further includes: determining, on the combined image, an orientation of the Frankfurt horizontal of a head encompassing the face relative to a horizontal and an orientation of the frontal plane of the head relative to a vertical; discarding the combined image if an angle between the Frankfurt horizontal of the head and the horizontal exceeds a third limit and / or if an angle between the frontal plane of the head and the vertical exceeds a fourth limit.

[0048] The orientation of the head encompassing the face can be determined from the combined image. This allows for a selection for further processing by using only those images in which the face is captured at a specific angle. This makes the determination of the probability value for each class more accurate. The orientation of the head can be determined, for example, using the method described under file number 10 2023 105 433.3.

[0049] According to one aspect, the method comprises: capturing a color image using a first camera designed to capture light with wavelengths in the visible range; and capturing a thermal image simultaneously with capturing the color image using a second camera designed to capture wavelengths in the infrared range; generating the at least one image pair by matching the color image and the simultaneously captured thermal image.

[0050] By matching the color image and the thermal image taken at the same time as the color image, the color image and the thermal image taken at the same time as the color image are linked together and thus form an image pair.

[0051] A color image is an image in the visible wavelength range. For example, a color image can be an RGB image in the RGB color space. R stands for a red color channel, G for a green color channel, and B for a blue color channel. In some aspects, a color image can be a CMYK image in the CMYK color space. C stands for a cyan color channel, M for a magenta color channel, Y for a yellow color channel, and K for a black color channel. In other aspects, a color image can be a monochrome image. A color image is a pixel-based, two-dimensional image, where the pixels are arranged in a grid. A camera designed to produce a color image captures wavelengths in the visible range, i.e., in a wavelength range of 400 nm to 700 nm, using a pixel-based sensor. The first type of camera can therefore be called a color image camera.

[0052] A thermal image can, for example, be a FLIR image (FLIR = Forward Looking Infrared). A FLIR image is a pixel-based, two-dimensional image, usually with one grayscale channel, where the pixels are arranged in a grid. A camera designed to capture a thermal image can also be called a thermal imaging camera. A thermal imaging camera captures wavelength ranges in the range of 3.5 µm to 15 µm, preferably 7 µm to 14 µm, using a pixel-based sensor. The second camera can also be called a thermal imaging camera. A thermal image can be an image in the infrared wavelength range.

[0053] The first and second cameras are preferably controlled such that they simultaneously capture a color image and a corresponding thermal image of the face. In other words, the thermal image and the color image can be captured at approximately the same time. "Simultaneously" in the context of this application means with a negligible time delay. For example, the thermal image and the color image can be captured with a time delay in the range of 1 ns to 10 ms. In some aspects, the thermal image and the color image can be captured with a delay of less than 10 ms. In other aspects, the thermal image and the color image can be captured with a delay of more than 1 ns.

[0054] An image pair can be generated, for example, by assigning the color image to a simultaneously acquired thermal image. This assignment can be achieved, for instance, by linking a first identification to the color image and a second identification to the thermal image acquired at the same time, with the first identification being linked to the second identification.

[0055] According to one aspect, the procedure includes: capturing a color image calibration image with the first camera of a test object with a plurality of light points; capturing a thermal image calibration image with the second camera of the test object with the plurality of light points; calculating, by the computer, the coordinate transformation between a perspective of the first camera and a perspective of the second camera by comparing the positions of the plurality of light points in the color image calibration image and the thermal image calibration image.

[0056] The first and second cameras can be positioned slightly offset from each other. This can result in slight differences in the perspectives of the two cameras.

[0057] Since the relative positions of the first and second cameras are usually fixed, their perspectives can be aligned using a coordinate transformation, particularly affine mapping, for each detection of facial manipulation. This coordinate transformation, especially affine mapping, can be determined through a calibration process. For this purpose, a thermal image calibration image and a color image calibration image are captured of a test object with several contrasting light points, with the first and second cameras fixed in their intended positions. The light points are typically visible in both the thermal and color images. By superimposing the light points in both images, the coordinate transformation, specifically affine mapping, can be calculated to align the perspectives of both cameras.

[0058] Thus, the calibration process allows the coordinate transformation, in particular the affine mapping, to be determined, which is used to convert the perspectives of both cameras into one another. The calibration process can be repeated for different distances to the cameras and the test object.

[0059] Furthermore, the procedure may in some aspects include the following steps: selecting the color image if the classification as to whether facial manipulation is present reveals that no facial manipulation is present; or discarding the color image if the classification as to whether facial manipulation is present reveals that facial manipulation is present; and generating image data using the selected color image.

[0060] This could create a method for determining image data that represents a digital biometric passport photo for a security document.

[0061] In some aspects, the procedure further includes: transferring the image data to a personalization device configured to personalize a security document, such as an identification card. Such a personalization device may, for example, include a printing unit that prints the color image onto the security document. The procedure may therefore also include: the personalization device printing the color image onto a security document based on the image data. Alternatively or additionally, the procedure may include: the personalization device storing the image data in a memory module of the security document. This personalizes the security document and enables identification of the individual based on the image data or the selected color image.

[0062] Such a procedure and such a personalization device is exemplified in European application EP 4 099 281 A1, the content of which relating to the personalization device and / or the procedure for printing a security document and / or the procedure for determining passport photo data is included herein.

[0063] In other aspects, the procedure also includes performing a facial recognition process using the image data.

[0064] Furthermore, the aforementioned problem is solved by a device, in particular a computer, for detecting facial manipulation, which is set up and designed to carry out the method described above, and a computer program product comprising instructions which, when executed by a data processing device, in particular the device, cause it to execute the process steps of the method described above.

[0065] Furthermore, the aforementioned problem is solved by a system for detecting facial manipulation, the system comprising: a device as previously described; a first camera associated with the device, wherein the first camera is designed to capture light with wavelengths in the visible range and to produce the color image; a second camera associated with the device, wherein the second camera is designed to capture wavelengths in the infrared range and to produce the thermal image; wherein the device is further configured and designed to control the first camera and the second camera such that the first camera captures the color image of the face and the second camera captures the thermal image of the face simultaneously.

[0066] Such a system can detect facial manipulation.

[0067] The first camera and the second camera can be aligned in the system such that their fields of view essentially overlap in one plane at a predetermined distance from the two cameras and have an essentially identical section in that plane.

[0068] In one aspect, the system includes means that are set up and designed to enable a person to independently capture the color image and the thermal image.

[0069] Such a means could, for example, be a trigger connected to the previously described device in such a way that, when the trigger is activated, a signal is sent to the device, causing a thermal image and a color image to be captured. The trigger could, for example, be positioned at least in one direction of the fields of view of the first and second cameras, so that a person can independently capture a corresponding thermal image and a corresponding color image of their face by activating the trigger.

[0070] Such a system can also be called a self-enrollment system. This system is specifically designed to allow a person to identify themselves independently without the presence of an authorized person, and to detect any manipulation. Furthermore, such a system can be the aforementioned personalization device.

[0071] The first camera and the second camera are installed in the system in such a way that the field of view of the first camera and the field of view of the second camera are identical at a predetermined distance. List of characters

[0072] Further properties, features and advantages of the invention will be clarified below by describing preferred embodiments of the invention with reference to the accompanying exemplary drawings, in which: Fig. 1 shows an exemplary schematic block diagram for a method for detecting facial manipulation; and Fig. Figure 2 shows an exemplary schematic representation of a system for detecting facial manipulation. Description of exemplary implementations

[0073] The features disclosed in the foregoing description, figures and claims can be important for the realization of the invention in its various embodiments, both individually and in any combination.

[0074] Reference symbols in the figures refer to the same elements.

[0075] Fig. Figure 1 shows an example of a schematic block diagram for a method for detecting facial manipulation.

[0076] In step S1, a calibration step is calculated to determine a coordinate transformation, specifically an affine transformation such as an affine mapping, between the perspective of the first camera 2, which produces a color image, and a second camera 3, which produces a thermal image. This step is performed in advance for a system with the first and second cameras 2, 3 being permanently installed and does not need to be repeated before each detection of facial manipulation. Step S1 is therefore optional.

[0077] For calibration, a color calibration image of a test object with multiple light points at a predetermined distance from the first camera (2) is captured with the first camera (3). Simultaneously, a thermal calibration image of the same test object with multiple light points at the same predetermined distance is captured with the second camera (3). The light points in both images are superimposed, thus determining a perspective shift between the two captured images. From this, an affine mapping is calculated for transforming the perspective between the thermal image and the color image.

[0078] In step S2, a color image in the visible wavelength range of the face of a person 4 to be identified is captured with the first camera 2. Simultaneously, a thermal image in the infrared wavelength range of the face of the person 4 to be identified is captured with the second camera 3. The face of the person 4 to be identified is located at a predetermined distance from the first and second cameras 2, 3. The color image and the simultaneously captured thermal image are combined so that both images form an image pair.

[0079] In step S3, specific facial features, such as eyes, nose or mouth, are recognized in the color image using artificial intelligence.

[0080] In step S4, the color image is scaled to a predetermined distance, for example on the order of 100 pixels, between the eyes of the face in the color image.

[0081] In step S5, the perspective of the thermal image is transformed to match the perspective of the color image using coordinate transformation, specifically affine mapping. The thermal image is then scaled to the same dimensions as the color image. The thermal image and the color image are thus essentially congruent. In particular, the face in the color image is essentially congruent to the face in the thermal image.

[0082] In step S6, the scaled color image and the scaled thermal image are cropped so that the face is positioned at a predetermined location within each image. This cropping can be performed, for example, based on ICAO rules. Cropping the face improves the analysis. However, the subsequent analysis can also be performed without cropping. Step S6 is therefore optional.

[0083] In step S7, the scaled color image and the scaled thermal image are combined, for example, by stacking them. The combined image thus generated preferably corresponds essentially to a tensor that links the information of the color image and the thermal image.

[0084] In a further optional step, the orientation of the head encompassing the face can be determined in the combined image. If the head exhibits a tilt exceeding a threshold, the corresponding image can be discarded and the process (S2-S7) repeated with a different image pair.

[0085] In step S8, a probability value is determined for each class specifying a type of manipulation using a convolutional neural network, indicating the probability that this type of manipulation is present.

[0086] In step S9, the probability values ​​for several classes representing a type of manipulation are normalized using a softmax function. A normalized probability value is a value between 0 and 1.

[0087] In step S10, the normalized probability values ​​for several classes indicating a type of manipulation are summed. If the sum of the normalized probability values ​​exceeds a first threshold, it is classified as facial manipulation overall. If the sum of the normalized probability values ​​falls below the first threshold, or below a second threshold that is lower than the first, it is classified as not manipulation.

[0088] Fig. Figure 2 shows an example of a schematic representation of a system for detecting facial manipulation.

[0089] Such a system includes a computer 1, which is designed and configured to perform the tasks associated with Fig. 1. To carry out the described procedure.

[0090] The system includes a first camera 2 connected to computer 1. The first camera 2 is designed to capture a color image in the visible wavelength range. Furthermore, a second camera 3 is connected to computer 1. The second camera 3 is a thermal imaging camera designed to capture a thermal image in the infrared wavelength range.

[0091] The first camera 1 and the second camera 2 are fixed within the system at a defined position and orientation relative to each other. The first camera 2 has a first field of view (represented by the corresponding dashed lines), and the second camera 3 has a second field of view (represented by the corresponding dashed lines). The first and second cameras 2, 3 are aligned such that their fields of view are essentially congruent in a plane parallel to the cameras at a predetermined distance within a section 5. The face of a person 4 to be identified is positioned within this section 5. Due to the alignment of the first and second cameras 2, 3, section 5 of the color image of the face of the person 4 to be identified corresponds to section 5 of the thermal image of the face of the person 4 to be identified.

[0092] Furthermore, computer 1 is configured to control the first and second cameras 2, 3 in such a way that the color image and the thermal image are captured simultaneously, i.e., with a negligible time delay. Computer 1 is also configured to identify the simultaneously captured images and combine them to create an image pair.

[0093] The computer 1 is further designed to perform the inventive method for detecting facial manipulation on the image pair.

[0094] The features disclosed in the foregoing description, the claims and the drawing can be important for the realization of the various embodiments, both individually and in any combination. List of reference symbols: 1 computer 2 First camera 3 Second camera 4 Person 5 Excerpt S1-S10 Procedure steps QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] EP 4 099 281 A1

[0062]

Claims

[1] A method for detecting facial manipulation, the method comprising the following steps carried out by means of a computer (1): - Receiving at least one image pair comprising a color image of a face and a simultaneously recorded thermal image of the face; - Adjusting the color image and the thermal image so that the face in the color image is congruent with the face in the thermal image; - Creating a combined image by combining the adjusted color image and the adjusted thermal image; - determining, on the combined image, for one or more classes indicating a type of manipulation, a probability value which indicates a probability that manipulation is present for the corresponding class; and - Classify whether facial manipulation has occurred based on the probability values. [2] The method of claim 1, wherein adjusting the color image and the thermal image comprises the steps of: - Determining characteristic facial features of the face in the color image; - scaling dimensions of the color image such that a distance between two of the characteristic facial features in the color image has a predetermined value; - perspective transforming the thermal image by means of a coordinate transformation so that a perspective of the thermal image and a perspective of the color image match; and - Scaling dimensions of the thermal image to the dimensions of the scaled color image such that the face in the scaled thermal image is congruent with the face in the scaled color image. [3] The method of claim 2, further comprising: - cropping the scaled color image based on the characteristic facial features so that the face is arranged at a predetermined position and with a predetermined size in the color image; - Cropping the scaled thermal image analogously to cropping the color image, so that the cropped thermal image and the cropped color image have the same section. [4] Method according to one of claims 2 or 3, wherein the characteristic facial features in the color image are determined by means of artificial intelligence. [5] A method according to any one of the preceding claims, further comprising: - Normalizing, by the computer, the probability value for each of the one or more classes to a normalized probability value; and - where classifying whether facial manipulation has occurred includes: - Summing the normalized probability values for several classes, - classifying that facial manipulation has occurred if the sum of the normalized probability values is greater than a first threshold, and - Classify that no facial manipulation is present if the sum of the normalized probability values is less than the first threshold or less than a second threshold that is lower than the first threshold. [6] Method according to one of the preceding claims, wherein the determination of a probability value for one or more classes indicating a type of manipulation is carried out by means of a convolutional neural network. [7] A method according to any one of the preceding claims, further comprising: - determining, by the computer, on the combined image an orientation of the Frankfurt horizontal of a head encompassing the face relative to a horizontal and an orientation of the frontal plane of the head relative to a vertical; - Rejecting the normalized image if an angle between the Frankfort horizontal of the head and the horizontal exceeds a third threshold and / or if an angle between the frontal plane of the head and the vertical exceeds a fourth threshold. [8] A method according to any one of the preceding claims, the method further comprising: - recording a color image by means of a first camera (2) which is designed to record light with wavelengths in the visible range; and - Recording a thermal image simultaneously with the recording of the color image by means of a second camera (3) which is designed to record wavelengths in the infrared range; - generating, by the computer (1), the at least one image pair by associating the color image and the thermal image recorded at the same time. [9] The method of claim 8, further comprising: - taking a color calibration image with the first camera (2) of a test object having a plurality of light points; - taking a thermal image calibration image with the second camera (3) of the test object with the plurality of light points; - Calculating, by the computer (1), the coordinate transformation between a perspective of the first camera (2) and a perspective of the second camera (3) by comparing the positions of the plurality of light points in the color image calibration image and the thermal image calibration image. [10] Device, in particular a computer (1), for detecting facial manipulation, wherein the device is arranged and designed to carry out the method according to one of claims 1-7. [11] Computer program product comprising instructions which, when executed by a data processing device, in particular the device according to claim 10, cause the device to carry out the method steps according to at least one of claims 1 to 7. [12] A system for detecting facial manipulation, the system comprising: - a device according to claim 10; - a first camera (2) in connection with the device, wherein the first camera (2) is designed to record light with wavelengths in the visible range and to generate the color image; - a second camera (3) in connection with the device, wherein the second camera (3) is designed to record wavelengths in the infrared range and to generate the thermal image; - wherein the device is further configured and designed to control the first camera (2) and the second camera (3) such that the first camera (2) records the color image of the face and the second camera (3) records the thermal image of the face at the same time as the color image. [13] The system of claim 12, wherein the system comprises means arranged and designed to enable a person to independently capture the color image and the thermal image.

Images