BIOMETRIC DOUBT REMOVAL METHODS AND DEVICES

Abstract

We describe an image processing method for generating a probability of the presence of a living person in an environment using the following steps: acquiring an image, called thermal data, of the person in an environment in the thermal infrared spectral range using matrix thermal sensors, acquiring a second image, called visible data, of the person in the visible spectral range using a CMOS imaging sensor, also capable of delivering data in the near-infrared range, then called near-infrared data, and generating this probability using a neural network, preferably a convolutional neural network having two convolutional layers, two subsampling layers and a fully connected layer whose inputs are the thermal data which can be supplemented by the visible data and / or the near-infrared data.

Description

Title of the invention: Title of the invention

[0001] METHODS AND DEVICES FOR REMOVING BIOMETRIC DOUBT.

[0002] The present invention relates to the field of image processing and more particularly to the electronic image capture methods and devices implemented in order to verify that the face of a living person is present in front of a camera device when taking a photograph, for example and without this being limiting, for the production of a passport.

[0003] The various falsification techniques possible when taking a picture of a person's face can be, for example: mask, paper print, telephone, computer or tablet screen representing a human face and placed in front of said shooting device.

[0004] However, if the shots are intended for the creation of official documents, such attempts must be detected in order not to create “false papers”. Advantageously, the solution must allow this detection regardless of the face of the person present in front of the detection device and therefore without the personal characteristics of this person having been previously recorded in a database.

[0005] Patents US2002 / 0136435Al and WO2010 / 0128931A1 disclose methods which can be used to create a device for detecting this type of falsification: two shooting systems are used for this.

[0006] The first system comprises a camera for taking pictures in the visible range of electromagnetic waves and the second system comprises another camera for taking pictures in the infrared range of electromagnetic waves.

[0007] The techniques implemented in these patents, in order to detect an attempt at falsification, require photographs with sufficient resolution to reliably identify characteristic elements of a human face: nose, ears, eyes, hair, among other examples. However, if in the visible domain this type of camera is common and costs less than a hundred euros, in the thermal infrared domain a resolution even of the "video" type, i.e. approximately 640x480 pixels, entails a high cost, for example of the order of several thousand euros.

[0008] The methods described in patent WO2006 / 003612A1 which operate a fusion of the images of the visible domain and the infrared domain also require that the definitions of the two images be compatible: if the definition of one of the images is much lower than the other, then the fusion of the images destroys all or part of the information relevant for the detection of the attempted falsification, information which was nevertheless contained in the higher definition image.

[0009] Furthermore, maintenance problems exist with the infrared sensors disclosed in these patents: in fact, these infrared sensors use micro-bolometers which require, before their first use, an initial calibration of each of the image elements (pixels) of the camera producing the images in the infrared domain and then re-calibrations in order to ensure their long-term operating stability.

[0010] To do this, it is necessary to call on a calibration laboratory which also uses expensive elements such as black bodies which are themselves calibrated or even, in addition, precise mechanical shutters.

[0011] Furthermore, patents WO2015 / 040001 and US 10,169,671 add to the previous methods a step of measuring the distance of the elements of the face, carried out using a complementary distance measuring system, or even the measurement of the entire head in the case of patent WO2015 / 040001, and this in order to mainly detect attempts at falsification using screens or printed documents which are therefore flat.

[0012] It goes without saying that the addition of these complementary systems further increases the cost of the falsification detection device and, moreover, they are not capable of effectively detecting an attempt at falsification by mask.

[0013] In patent US9,230,158B 1, a method for detecting falsification is described based on the study of variations or the absence of variations in skin tone from images taken from a video sequence. This approach, however, faces the following problems: if this technique is capable of detecting the absence of changes in skin tone of a fixed image over time, when the attempt at falsification is made from a video sequence of the person and no longer from a fixed image, then the system can more difficultly detect it because the variations in skin tone during the sequence have actually been recorded.Furthermore, since the variations in skin tone within a few seconds are very small, or even invisible to the naked eye as stated in the patent itself, the method in question requires the use of imaging devices with a very high signal-to-noise ratio and therefore, again, high cost. The method disclosed by patent WO2014 / 004179Al, which also analyzes variations in skin tone, faces the same problem.

[0014] On the other hand, new alternative techniques for viewing images in the thermal infrared domain are now available.

[0015] These new techniques use elements made up of thermopiles as infrared sensors and no longer micro-bolometers as in all the patents presented previously working in the thermal infrared field. They no longer require re-calibration. They are also much less expensive: only a few dozen euros.

[0016] In return, these devices have a very limited definition, currently less than 1000 image elements (pixels) and therefore do not provide sufficient resolution to identify the facial elements necessary for carrying out the method described in patent US 2002 / 0136435Al and even less to authorize the identification of people by comparison with a database for example.

[0017] The methods described in patent WO2017 / 070920Al also use a database of previously recorded images of people. However, for the invention presented, the context is completely different: it concerns methods and devices for verifying that the face of the person being tested corresponds to someone living without data relating to this person having been previously recorded in any way whatsoever in a database. In addition, patent WO2017 / 07092Al uses wavelengths corresponding to the near infrared: 800nm ​​- 1800nm, values ​​recalled in the description of the patent, whereas the temperature of human skin typically falls on a scale of values ​​from approximately 28°C to 38°C, it would be advantageous to use infrared wavelengths in a range of 5000 nm to 25,000 nm.

[0018] However, precisely, another advantage of new infrared thermopile visualization techniques is that the wavelengths concerned are located in particular in the thermal infrared (4000 nm - 15000 nm) and no longer in the near infrared (wavelength between 800 and 2000 nm): in this way, it is also unnecessary to incorporate into the detection device an infrared illuminant external to the face under test as is for example the case in the descriptions of patents US 2006 / 0104488, US 7,147,153B2 or EP1533744A2.

[0019] Patent US10,169,671B2 mentions the use of such temperature measuring devices by a matrix of elements sensitive to infrared wavelengths. On the other hand, in the method described, a single temperature value is used: in the method described, the matrix element spatially corresponding to the center of the face under test is interrogated. The changes in temperature on the different elements of the measurement matrix are therefore not taken into account in the method.

[0020] Patent WO2018 / 0558554A1 describes other techniques for detecting falsification attempts using neural networks.

[0021] In this patent, only information from shots in the visible domain is used. As mentioned previously, the resolution of shots in this domain achieved by devices costing only a few dozen euros is sufficiently high to allow the identification of people, for example, by comparing the data obtained by these devices with those recorded in an image database. Such a comparison is used, for example, to authorize access to a room, a computer, a telephone, etc. However, here, we are seeking to detect the attempted falsification before identification using the smallest possible means and without comparison with a database. In addition, the method describes the use of two neural networks, one working on the spatial domain and another on the temporal domain: it is therefore necessary to record a complete sequence of visible images, therefore of high resolution, described in this patent as a video stream: the amount of memory used by a device implementing the method increases significantly compared to a method implementing a smaller number of images or even a single image. Furthermore, the method describes the use of two neural networks: this increases the calculation time necessary to carry out the method and therefore greatly impairs the responsiveness of the device.In addition, the computation time of the training phase of the two neural networks is, again, doubled compared to a single neural network, which increases the implementation cost.

[0022] Finally, since this method uses information in the visible domain only, it is easier for a falsifier to deceive, which, again, greatly reduces the interest of the method described.

[0023] The present invention makes it possible to solve the problems mentioned above.

[0024] To this end, a method according to a first aspect of the invention comprises the steps consisting of:

[0025] acquiring data representative of an image in the thermal infrared spectral domain, called thermal data, of a person in an environment by means of a matrix sensor,

[0026] and is essentially characterized in that it further comprises the following steps:

[0027] storing these thermal data,

[0028] transmit this thermal data to a processor,

[0029] process this thermal data by a processor,

[0030] generate a quantity representative of the probability of the actual presence of a living person using a neural network whose inputs are thermal data.

[0031] In preferred embodiments, one and / or the other of the following arrangements are used in which:

[0032] - a step is planned to acquire data representative of the face of the person to constitute the thermal data,

[0033] - a step is provided to acquire data representative of the same person, called visible data, in the visible spectral domain by means of a camera, the neural network having in addition the visible data among its inputs,

[0034] - a step is provided for acquiring data representative of the face of the person to constitute the visible data,

[0035] - a step is provided for using data representative of a reconstruction in three dimensions of the person in the environment,

[0036] - a step is provided for acquiring data representative of an image, called near infrared data, of the person in the near infrared spectral range using the same camera, the neural network having in addition the near infrared data among its inputs,

[0037] - we use a convolutional neural network,

[0038] - we use a convolutional neural network comprising two layers of convolution, two downsampling layers and one fully connected layer,

[0039] - a step is provided to acquire thermal data over time for constitute the inputs of the neural network,

[0040] - a step is planned to acquire the visible data over time for constitute the inputs of the neural network,

[0041] - a step is planned to acquire near infrared data over time to constitute the inputs of the neural network,

[0042] - a step is provided to acquire thermal data over time for generate a quantity representative of the probability of the person moving in the environment,

[0043] - a step is provided to resize the size of the visible data and / or the near infrared data at the size of thermal data,

[0044] - a step is provided to introduce a time shift between the data thermal and visible data.

[0045] A second aspect of the invention relates to a system comprising:

[0046] a system for capturing images, arranged so as to enable the capturing of images in the thermal infrared range of a person in an environment, the capturing means providing digital data representative of the images called thermal data,

[0047] and is essentially characterized by:

[0048] means for storing thermal data,

[0049] means for transmitting thermal data,

[0050] exploitation means for exploiting this thermal data,

[0051] the operating means comprising a neural network arranged to generate a quantity representative of the probability of the actual presence of a living person, the inputs of the neural network being the thermal data.

[0052] In preferred embodiments, one and / or the other of the following arrangements are used:

[0053] - the image capture system is arranged so as to allow the capture of images in the thermal infrared range of the person's face,

[0054] - the shooting system is also arranged so as to allow the taking images of the person in the visible domain, the means of taking pictures providing digital data representative of the images called visible data, the neural network also having the visible data among its inputs,

[0055] - the shooting system is also arranged so that the means exploitation uses data representative of a three-dimensional reconstruction of the person in the environment,

[0056] - the image capture system is arranged so as to allow the capture of images in the visible and / or near infrared range of the person's face,

[0057] - the image capture system is arranged to allow images to be taken of the person in the near infrared range, the shooting means providing digital data of the images called near infrared data, the neural network having in addition the near infrared data among its inputs,

[0058] - the neural network is arranged as a convolutional neural network,

[0059] - the neural network is arranged as a convolutional neural network comprising two convolution layers, two downsampling layers and a fully connected layer,

[0060] - the neural network also receiving on its inputs the evolution of the data thermal over time,

[0061] - the neural network also receiving on its inputs the data visible to several successive moments,

[0062] - the neural network also receiving infrared data among its inputs close over time,

[0063] - the exploitation means are arranged so as to exploit the data thermal and to generate a quantity representative of the probability of movement of a person in the environment over time,

[0064] - the operating means are arranged to resize the size of the data visible and / or near infrared data at the size of thermal data,

[0065] - the operating means are arranged so as to introduce an offset time difference between thermal data and visible data.

[0066] The invention comprises, apart from these main arrangements, certain other arrangements which are preferably used at the same time and which are discussed in more detail below.

[0067] In the following, some preferred embodiments of the invention will be described with reference to the attached figures in a manner which is of course not limiting.

[0068] In the drawings,

[0069] Figure [Fig.l] is a schematic view of a device according to the invention.

[0070] Figure [Fig.2] is a decomposition of the neural network according to the invention.

[0071] A system is described that is set up for taking pictures of people (3) comprising a camera system (2) arranged to take photographs of these people present in the environment (1) and the aim of which is to detect an attempt to falsify the photograph of the person (3) using means such as masks, prints or screens presented to the camera system (2).

[0072] In one embodiment of the invention, the image capture system is arranged to take pictures of the face (30) of living persons (3) whose position in the environment (1) is known, for example the person (3) is seated on a seat (4) in the middle (1).

[0073] In another embodiment of the invention, the shooting system is arranged to take pictures representing a scene larger than the face (30) of the people (3) moving in the environment (1).

[0074] In a manner known per se, the shooting system or camera (2) comprises at least one processor (21), for example a Cypress CYUSB3065-BZXI equipped with an internal volatile memory (211) which can be initialized for example by means of a USB connection (26) and a personal computer or "PC" (27). The program code (210) necessary for the operation of the processor (21) is here executed from the memory (211). A non-volatile memory (23) can be connected to the processor (21) to store additional information such as operating parameters of the camera (exposure time, gain, resolution among other non-restrictive examples), the program code (210) when the operation of the chosen processor (21) requires it. Similarly, an additional volatile memory (25) can be connected to the processor (21) when the operation of the chosen processor (21) and / or its program code (210) requires it.

[0075] The camera (2) also comprises a thermal imaging sensor (22) sensitive in the thermal infrared range, for wavelengths between 4000 nm and 15000 nm. The sensor (22) delivers images in the form of digital data, hereinafter called thermal data and is normally provided with an optic (220). The sensor (22) is connected in a manner known per se to the processor (21), via a connection (28), this connection being able to be, for example, a direct parallel connection between the pins of the processor (21) and those of the sensor (22) or a serial connection between the pins of the processor (21) and those of the sensor (22) such as I2C, SPI, mipi, Ivds among the possible embodiments.

[0076] In one embodiment according to the invention, the sensor (22) is a Melexis MLX90640ESF-BAB-000 thermal infrared imaging sensor: it comprises a matrix (221) of 32x24 image elements (pixels) and the optics (220) integrated into the sensor (22) having a field of view of 55°x40° is therefore suitable for taking pictures of the face (30).

[0077] In another embodiment according to the invention, the sensor (22) is a Melexis MLX90640ESF-BAA-000 thermal infrared imaging sensor: it comprises a matrix (221) of 32x24 pixels and the optics (220) integrated into the sensor (22) having a field of view of 110°x75° is therefore suitable for taking pictures of the person (3) moving in the environment (1).

[0078] In these two embodiments, the sensor (22), whether the Melexis MLX90640ESF-BAB-000 or the Melexis MLX90640ESF-BAA-000, further comprises a memory, thus enabling it to store the thermal data.

[0079] The program code (210) of the processor (21) can then use the connection (28) to reread this memory of the sensor (22) in a delayed manner, and in this way, the processor (21) requires a much shorter processing time compared to that which would be required for real-time reading of the thermal data.

[0080] Indeed, the program code (210) of the processor (21) can request the reading of the memory of the sensor (22) at any time as long as thermal data is stored and the processor (21) remains in complete control of this time and the rate at which this reading request is sent.

[0081] On the contrary, if the reading of the thermal data is carried out in real time, then the processor (21) is totally a slave to the sensor (22) and must permanently process the thermal data transmitted by said sensor (22) so that this thermal data is not corrupted: the processor (21) is then interrupted regularly and must process this thermal data as quickly as possible, which reduces the time available for executing the program code (210).

[0082] Finally, in a particular embodiment, a serial I2C type connection (28) uses only two pins of the processor (21) to communicate with the sensor (22) and said processor (21) therefore does not use its pins dedicated to the transmission of video signals, such as mipi pins among other examples, for this connection (28).

[0083] Therefore, the processor (21) can both be of lower cost than a processor which would have two sets of pins dedicated to the transmission of video signals from two different imaging sensors and advantageously use a single set of pins dedicated to the transmission of video signals to communicate with an imaging sensor other than the thermal imaging sensor.

[0084] Thus, the camera (2) may also comprise another imaging sensor (24), in particular sensitive in the visible range. The sensor (24) delivers images in the form of digital data, hereinafter called visible data, and is normally provided with an optic (240). In a manner known per se, the sensor (24) is connected to the processor (21), for example via a direct parallel connection, or via a serial connection such as I2C, SPI, mipi, Ivds among the possible embodiment examples.

[0085] The image sensor (24) is for example an Omnivision OV5645 component (241) delivering images with a resolution of up to 2592x1944 image elements or pixels in the form of digital data, and normally provided with an optic (240), for example with an Ml2 mount, adapted to the size of said sensor.

[0086] In one embodiment according to the invention, the optic (240) is chosen so as to display a field of view adapted to shots of the face (30): for example, an M12 Sunex DSL945D-650-F.25 optic, compatible with the Omnivision OV5645 component (241) and whose horizontal field of view is 55°.

[0087] In another embodiment according to the invention, the optic (240) is chosen so as to display a field of view adapted to taking pictures of the person (3) moving in the environment (1): for example, an M12 Sunex DSL377A-650-F2.8 optic, compatible with the Omnivision OV5645 component (241) and whose horizontal field of view is 113°.

[0088] In a particular embodiment according to the invention, the acquisition times of the sensor (22) and the sensor (24) are temporally offset from each other by a value of dt seconds, dt being a real value for example between -10 seconds and + 10 seconds, a positive value of dt meaning that the thermal data are acquired before the visible data while a positive value of dt meaning that the visible data are acquired before the thermal data.

[0089] Therefore, the method is even more difficult to deceive by the presentation of falsified images, the falsifier not being able to determine when to present these images due to this shift dt, this value dt itself being able to be random by the use in the program code (210) of a random number generator, known per se, for example a linear congruential Lehmer generator.

[0090] In another particular embodiment according to the invention, the camera (2) comprises another image sensor, preferably identical to the image sensor (24).

[0091] Using, for example, the methods described in the document Foundations and Trends in Computer Graphics and Vision, Vol. 4, No. 4 (2008) p 287-404 by T. Moons, L. Van Gool and M. Vergauwen, the visible image data from the sensor (24) and the visible image data from this additional image sensor can be combined by the computer (27) which then performs a 3-dimensional reconstruction operation of the medium (1).

[0092] Additional information, such as an estimate of the volume of the person's head (3), directly accessible by this 3-dimensional reconstruction operation dimensions, can then be exploited by the computer (27), excluding the possibility for a falsifier to present flat images.

[0093] When the processor (21) does not have enough pins to simultaneously connect two image sensors such as the image sensor (24), an additional electronic component is inserted between the different image sensors and the processor (21).

[0094] This additional electronic component may, for example, be a programmable component (FPGA) such as a Lattice LIF-MD6000-6MG81I, costing around €10, equipped with the dedicated “Crosslink” functionality making it possible to connect 2 or even 4 image sensors such as the image sensor (24) to a single processor (21).

[0095] A SPI serial type memory such as an Adesto AT45DB321D, which can be programmed in a step prior to carrying out the method according to any mode known per se, for example using an EETools TopMax2 programmer, and containing a program code equipped with the “Crosslink” functionality is then present on the camera and connected to the programmable component using their pins dedicated to this use.

[0096] Furthermore, when the image sensor (24) used is in particular of the CMOS imager type, then this sensor (24) can also be sensitive in the near infrared range, typically defined here by wavelengths of 700 nm to 1000 nm.

[0097] However, for the Omnivision OV5645 component, with wavelengths between 800 nm and 1000 nm, all the image elements of the component have a substantially identical response and, therefore, this component has the behavior of a grayscale image sensor, which implies for these wavelengths an essentially luminance response.

[0098] In practice, so that the sensor (24) has a response in the visible range only, the wavelengths of which are typically between 400 nm and 700 nm, the optics (240) or the M12 mount contains a bandpass optical filter.

[0099] In a first exemplary embodiment, an M12 mount of the FLIR ACC-01-5001 type itself contains the optical bandpass filter.

[0100] In a second embodiment, the Sunex DSL945D-650-F2.5 and DSL377A-650-F2.8 optics (241) can be ordered from their manufacturer with optical processing integrating the optical bandpass filter.

[0101] In another particular embodiment according to the invention, several different filters, bandpass or other, can be placed in front of the component (241) during operation: for example, by using an M12 mount designed for this purpose.

[0102] Thus, an M12 Scorpion Vision Limited BR-M12 mount can contain several filters, namely, a first visible bandpass filter and a second bandpass filter near infrared, the selection then being carried out by an electronic control connected to the processor (21) via one of its general purpose interface pins (GPIO).

[0103] In this particular embodiment, if the second bandpass filter is activated by the processor (21) and the optical frequency band selected by said filter corresponds to the operation of the image sensor (241) in gray levels, the sensor (24) then delivers images in the near infrared spectral domain in the form of digital data, called near infrared data hereinafter.

[0104] These near infrared data have, by construction, the same definition as the visible data.

[0105] In a preferred embodiment according to the invention, the processor (21) communicates the thermal data and the visible data and / or near infrared data to the main processor (271) of the computer (27) by means of the USB connection (26).

[0106] In another embodiment according to the invention, the processor (21) communicates to the computer (27) the thermal data and the visible data and / or near infrared data by means of a wired or wireless connection: Ethernet, WiFi, Bluetooth, Zigbee or by any other wired or wireless connection known per se.

[0107] In another embodiment according to the invention, the processor (21) itself is programmed to carry out the processing of thermal data and visible data and / or near infrared data.

[0108] A neural network (270) set up for the invention is now described as the constitution of one or more layers of neurons interconnected with each other.

[0109] A neuron is associated with a transfer function subsequently called synaptic weight between all its inputs and its output.

[0110] A layer of neurons represents the juxtaposition of several neurons sharing the same inputs, each therefore having its own synaptic weight and its own output.

[0111] Each neuron of a layer can thus be connected to one or more neurons of one or more preceding layers, the last layer being the “output layer” of the neural network (270), i.e. in the embodiment according to the invention, a quantity representative of the probability of presence noted PR of a person (3) living in the environment (1).

[0112] The synaptic weights of the neural network (270) therefore characterize its operation and the optimal values ​​of these weights are determined during a step prior to the implementation of the method according to the invention, called learning.

[0113] The synaptic weights have a real, unbounded value.

[0114] In a preferred embodiment of the invention, the neural network (270) used by the method according to the invention is a convolutional neural network which, by definition, is made up of at least one convolution layer, at least one sub-sampling layer commonly called “Pooling” layer and at least one fully connected layer, each of whose functions will be explained.

[0115] An output of each layer of the neural network (270) is produced by all the outputs of the neurons composing said layer and is subsequently called an output map.

[0116] Similarly, the input map of a layer can either be the output map of a previous layer or all or part of the signals to be processed by the neural network (270).

[0117] In a preferred embodiment according to the invention, the input map of the neural network (270) is constituted by the only thermal data on which a normalization step is applied by the processor (271), known per se, so that these thermal data vary over a range [Vamin,Vamax]; Vamin representing the lowest temperature and Vamax the highest temperature. For example, if the medium (1) is located inside a building, Vamin can represent a temperature recorded by an element of the sensor (22) of 15°C or less and Vamax represent a temperature recorded by an element of the sensor (22) of 40°C and more and it is then possible to use an affine function mathematically to carry out this normalization step.

[0118] In one embodiment according to the invention, the method uses a set of values ​​Vamin Vamax such as: Vamin= 0 and Vamax= 1.

[0119] In another embodiment according to the invention, the method uses a set of values ​​Vamin Vamax such as: Vamin= -0.5 and Vamax= +0.5.

[0120] In another embodiment according to the invention, the input layer of the neural network (270) is completed by the visible data and / or the near infrared data on which a normalization step is applied by the processor (271), known per se, so that these visible data and / or near infrared data vary over a range of [Vbmin,Vbmax]; Vbmin representing the smallest value observable by the sensor (24) and Vbmax representing the largest value observable by the sensor (24). For example, if the visible data and / or near infrared data vary over a dynamic range of one byte, i.e. [0,255], then an affine function makes it possible, again, to mathematically carry out the normalization step.

[0121] In one embodiment according to the invention, the method uses a set of values ​​Vbmin, Vbmax such as: Vbmin= 0 and Vbmax= 1.

[0122] In another embodiment according to the invention the method uses a set of values ​​Vbmin Vbmax such as: Vbmin= -0.5 and Vbmax= +0.5.

[0123] Usually, the method preferentially uses values ​​such as: Vamin = Vbmin and Vamax = Vbmax and in this way, the part of the input map corresponding to the thermal data and the part of the input map corresponding to the visible data and / or near infrared data have the same dynamics.

[0124] In a particular embodiment, it is useful to insert a step of resizing the visible data and / or near infrared data so that they have the same dimensions as the thermal data: in our exemplary embodiment the thermal data represents a matrix of 32x24 elements while the visible data and / or near infrared data represents a matrix of 2592x1944 elements.To perform this transformation from 2592x1944 to 32x24 elements, the processor (271) applies for example a sub-sampling; the simplest being to use a mathematical method consisting for example in selecting a value every n values, n being an integer for example n=81, in the horizontal dimension of the matrix of visible data and / or near infrared data and a value every p values, p being an integer for example p=66, in the vertical dimension of the matrix of visible data and / or near infrared data in order to form a new matrix of visible data and / or near infrared data comprising 32x24 elements for the matrix of visible data and / or 32x24 elements for the matrix of near infrared data.

[0125] Whatever the embodiment according to the invention chosen described previously, the input map (2700) of the neural network (270) is made up of a two-dimensional matrix set: in our exemplary embodiment, a first matrix of 32x24 elements where each of the elements has a range of values ​​between [Vamin,Vamax] representing the thermal data, a second matrix of 32x24 elements where each of the elements has a range of values ​​between [Vbmin,Vbmax] representing, if applicable, the visible data and / or another matrix of 32x24 elements where each of the elements has a range of values ​​again between [Vbmin,Vbmax] by construction representing, if applicable, the near infrared data.

[0126] In the preferred embodiment according to the invention, the matrix dimensions hxv of the input map representing the thermal data and the visible data and / or the near infrared data of the neural network (270) are h integer = 32 and v integer = 24.

[0127] It goes without saying that if the dimensions hxv of the matrices of the sensors (22) are different from those presented in the exemplary embodiments according to the invention, then the person skilled in the art uses new matrix sizes for the input card corresponding to these new matrix dimensions.

[0128] At the output of the previous step of the method according to the invention, the input map (2700) of the neural network (270) is thus mathematically represented in the form of matrices I(z)hv, z being an integer equal to 0 for thermal data, 1 for visible data and / or 2 when the near infrared data are used in a particular embodiment according to the invention.

[0129] Depending on the implementation options, we can therefore have among other possibilities: zE {0}, zE {0,1}, zE {0,2} and Z € {0,1, 2} for operation of the neural network respectively from only thermal data, thermal data and visible data, thermal data and near infrared data and finally thermal data, visible data and near infrared data.

[0130] We now explain the operation of the first convolution layer of the neural network (270).

[0131] The first convolution layer generates from a constituent element of the input map I(z)hjV one or more output maps (2701) Oi(z), each output map being defined by a mathematical convolution operation using a convolution kernel NC;(z) of size (ni,n2) comprising coefficients NC;(z,k,l), k being an integer varying from 1 to ni and 1 being an integer varying from 1 to n2, these coefficients NC;(z,k,l) ​​being the synaptic weights, for this first convolution layer, defined during the learning step which is applied to the elements I(z)h,v.

[0132] Convolution is, in the method according to the invention, the mathematical process consisting of adding each element of the input map to its local neighbors, weighted by the elements of the kernel.

[0133] In a preferred embodiment according to the invention, the convolution kernels NCi(z) are of size 3x3, i.e. ni = 3 and n2 = 3.

[0134] In a preferred embodiment according to the invention, the first convolution layer of the neural network (270) generates 2x4 output maps O;(z), with i being an integer between 1 and 4, z being between 0 and 2.

[0135] Mathematically, the values ​​located at each location (x,y) of the output map O;(z), with x being an integer varying from 1 to h and y being an integer varying from 1 to v are represented by the following formula:

[0136] OiW (xy) .x+k j+I} ,NC^ k „ ) +Blj(z) )

[0137] where af is a mathematical function of activation of the neuron,

[0138] Bli(z) is defined as a real, unbounded value, common to all neurons in the first convolution layer.

[0139] In a first exemplary embodiment according to the invention, the value Bl;(z) is zero.

[0140]

[0141]

[0142]

[0143]

[0144]

[0145]

[0146]

[0147]

[0148] In another exemplary embodiment according to the invention, the value Bl;(z) is determined during the learning step. In a preferred embodiment of the invention, the identity function is used. In another embodiment according to the invention af is the hyperbolic tangent mathematical function. In another embodiment according to the invention af is the following mathematical function: , x F xsix>0 1 X (0,lxsïx<0j The next step of the method according to the invention is carried out by a first “Pooling” layer of the neural network (270). The input map of this Pi Pooling layer consists of the output maps Oi(z) generated in the previous step. The output maps (2702) OP;(z) are generated from the maps O;(z) by the mathematical sub-sampling operations called “Max Pooling”, explained by the following formula: OP^z, x, y') = b f ( (max{ O^z, 2x', 2y')} ) +32^2) ) VkeNetl^k^min^iip h-2- x') Vl&Netl^l^min(n2, v-2y')

[0149] where N is the set of integers

[0150] X is an integer ranging from 1 to

[0151] y is an integer ranging from 1 to ,

[0152] and bf is the activation function of this step,

[0153] B2i(z) is defined as a real, unbounded value, common to all neurons in the first Pooling layer.

[0154] In a first exemplary embodiment according to the invention, the value B2;(z) is zero.

[0155] In another exemplary embodiment according to the invention, the value B2;(z) is determined during the learning stage.

[0156] In a preferred embodiment according to the invention, the synaptic weights of this step are all unitary and the activation function bf is the following mathematical function:

[0157] b f (x) xsix 0 _0,lxsïx< 0.

[0158] At the output of this Pooling layer Pb the matrices representing the output maps OPi(z) therefore comprise half as many rows and half as many columns as the output maps Oi(z) of the previous convolution step.

[0159]

[0160]

[0161]

[0162]

[0163]

[0164]

[0165]

[0166]

[0167]

[0168]

[0169]

[0170]

[0171]

[0172]

[0173]

[0174] The next step of the method according to the invention is carried out by a second convolution layer denoted R of the neural network (270). The input map of this new convolution layer R consists of the output maps OP;(z) generated in the previous step. The second convolution layer generates from a constituent element of the input map OPi(z)h / 2jV / 2 one or more output maps (2703) Rj(z), j being an integer greater than or equal to 1, each output map being defined by a mathematical convolution operation using a convolution kernel NDj(z) of size (pi,p2) comprising coefficients Ndj(z,k',l'), k' being an integer varying from 1 to pi and 1' being an integer varying from 1 to p2, these coefficients Ndj(z,k',l') being the synaptic weights, for this second convolution layer, defined during the learning step which is applied to the elements OPi(z)h / 2jV / 2. In a preferred embodiment according to the invention, pi = ni and p2 = n2 are used for the size of the convolution kernels NDj(z) and the method generates j= 2i output maps Rj(z). Mathematically, the values ​​located at each location (x”,y”) of the output map Rj(z), with x” an integer varying from 1 to h / 2 and y” an integer varying from 1 to v / 2 are represented by the following formula: Rj(z, x", y ") = cf ( ^OP^z, x"+k', j"+l')NDj(.z, k', P) ) +B3j(z) ​​) where Cf is a mathematical function of neuron activation, B3j(z) ​​is defined as a real, unbounded value, common to all neurons in the second convolution layer. In a first exemplary embodiment according to the invention, the value B3j(z) ​​is zero. In another exemplary embodiment according to the invention, the value B3j(z) ​​is determined during the learning step. In a preferred embodiment of the invention, the identity function is used. In another embodiment according to the invention cf is the hyperbolic tangent mathematical function. In another embodiment according to the invention cf is the following mathematical function: xsix^O Cf X (OzlxsixcO. The next step of the method according to the invention is carried out by a second “Pooling” layer of the neural network (270). The input map of this Pooling layer P2 consists of the output maps Rj(z) generated in the previous step.

[0175] The output maps (2704) RPj(z) are generated from the maps Rj(z) by the mathematical sub-sampling operations called “Max Pooling”, explained by the following formula:

[0176] RPj(z, x'“, y'") = d f ( (max{Rj(z, 2X 1 ", 2y'")} ) +B4j(z) ) V k eNetl^ k'c min^p^ h / 2-2x'"} VI g^min^p y vl2-2y“'j

[0177] where N is the set of integers

[0178] ^'"is an integer ranging from 1 to 0,

[0179] y'"is an integer ranging from 1 to

[0180] and df is the activation function of this step,

[0181] B4j(z) is defined as a real, unbounded value, common to all neurons of the second Pooling layer.

[0182] In a first exemplary embodiment according to the invention, the value B4j(z) is zero.

[0183] In another exemplary embodiment according to the invention, the value B4j(z) is determined during the learning stage.

[0184] In a preferred embodiment according to the invention, the synaptic weights of this step are all unitary and the activation function df is the following mathematical function:

[0185] xsix > 0

[0186] At the output of this Pooling layer P2, the matrices representing the output maps RPj(z) therefore includes half as many rows and half as many columns as the output maps Rj(z) of the previous convolution step.

[0187] The next step of the method according to the invention is carried out by the fully connected layer denoted CC (2704) of the neural network (270).

[0188] The input map of this new fully connected layer CC consists of the output maps RPj(z) generated in the previous step.

[0189] In this CC layer, a synaptic weight Ps is associated with each of the (h / 4) • ( v / 4) • (2• i) • k elements of the output maps RPj(z) with s number integer varying from 1 to {h! 4 ) • ( v / 4 ) • ( 2 • i) • k and j varying in the preferred embodiment from 1 to ( 2 • i ) the output result PR (2705) of the neural network (270) then being expressed as: [° 19 °] PR= e f (

[0191] where ef is a mathematical function of activation of the neuron,

[0192] B5 is defined as a real, unbounded value,

[0193] k being an integer, at least equal to 1, representing the number of elements of the variation range of z.

[0194] In a first exemplary embodiment according to the invention, the value B5 is zero.

[0195] In another exemplary embodiment according to the invention, the value B5 is determined during the learning stage.

[0196] In a preferred embodiment of the invention efest is a sigmoid mathematical function: [01^1 6^)=^

[0198] The output PR of the neural network (270) is thus between 0 and 1, and constitutes the quantity representative of the probability of the presence of a person (3) living in the environment (1) estimated by the neural network (270), that is to say a first result according to the invention.

[0199] All the mathematical operations to be carried out during the different steps of the method according to the invention relating to the operation of the neural network (270) being now explained, they are preferably carried out by the processor (271) of the computer (27), on one or more calculation cores of said processor (271) if it is itself made up of several of these cores and / or by a coprocessor (272) of the processor (271), by processors (2730) of the graphics card (273) of the computer (27) or any other element present in the computer (27) capable of carrying out all or part of the calculation operations to be carried out during the operation of the neural network: signal processing processor (DSP), dedicated programmable logic component (FPGA), expansion card pluggable into the computer (27) or any other element known per se capable of carrying out calculations.

[0200] In a particular embodiment of the invention, these calculation operations carried out by the neural network are carried out by the processor (21) of the camera itself, said processor communicating the calculation result PR to the computer (27) for example using the USB connection (26) or by any wired or wireless means of connection between this processor (21) and the computer (27).

[0201] The training of the neural network (270) can be carried out using a training base, consisting of thermal data and visible data and / or near infrared data corresponding to real people (3), previously recorded on memory media known per se (hard disk, USB key, SD memory card among other examples), data for which the expected output of the neural network is a probability of 1, i.e. absence of falsification, and is also consisting of thermal data and visible data and / or near infrared data corresponding to falsifications of people using masks, impressions or screens, previously recorded and for which the expected output of the neural network is a probability of 0, i.e. detection of falsification.

[0202] Learning is therefore the step prior to the method assigning values ​​to the synaptic weights as well as the elements Bl;(z), B2;(z), B3j(z), B4j(z), B5 when the embodiment according to the invention does not use a zero value for the latter and which consists of finding an optimum of all these values ​​from the learning base using an optimization method known per se.

[0203] Among the possible optimization methods, we can cite the back-propagation of the gradient, consisting, by using an input map extracted from the learning base to the neural network (270), in calculating the magnitude representative of the output probability provided, by the neural network (270) at this moment of its learning, in comparing the known output of the learning base, here 0 or 1 then in back-propagating an error signal in the neural network in order to correct the synaptic weights as well as the elements B1;(z), B2;(z), B3i(z), B4i(z), B5 when the embodiment according to the invention which has been chosen does not use a zero value, for example by means of a known gradient descent method.

[0204] When the difference between the probability PR delivered by the neural network (270) and the known result of the learning base is less than a reference value called criterion, represented by a real value noted ai, the learning step is considered finished and the synaptic weights can then be used for the implementation of the method according to the invention.

[0205] In one embodiment according to the invention, the learning step is considered complete when the following formula is valid for the N elements of the learning database:

[0206] \PR r -BD r \ ^a v

[0207] with N: integer greater than or equal to 1, representing the number of elements present in the learning base,

[0208] PRr: output map of the neural network for element r, r being an integer at least equal to 1 and less than or equal to N,

[0209] BDr: known result for an element r of the training database, i.e. 0 for an attempted falsification and 1 for a real person,

[0210] ai: acceptance criterion for the entire training database, ai being a real numerical value less than 1.

[0211] An acceptance criterion ai = 0.01 can be used in this embodiment. More stringent acceptance criteria can of course be chosen, for example ai = 0.001 but then the learning step is longer in duration.

[0212] In another embodiment according to the invention, the learning step is considered to be completed when the following formula is verified: 102131

[0214] with N: integer greater than or equal to 1, representing the number of elements present in the learning base,

[0215] PRr: output map of the neural network for the element r of the learning base, r being an integer at least equal to 1 and less than or equal to N,

[0216] BDr: known result for an element r, i.e. 0 for an attempted falsification and 1 for a real person,

[0217] ai: acceptance criterion for the entire training database, ai being a real numerical value less than 1.

[0218] An acceptance criterion ai = 0.01 can be used in this embodiment. More stringent acceptance criteria can of course be chosen, for example ai = 0.001 but then the learning step is longer in duration.

[0219] After this learning step, if new thermal data and visible data and / or near infrared data not appearing in the learning base constitute a new input map, then the neural network calculates a new quantity representative of the probability PR of the presence of a real person (3) in the environment (1).

[0220] Depending on the result PR, the main processor (271) determines whether there has been an attempt at tampering by comparing with a previously set threshold a2 PR: for example, a value of a20.6 can be used to confirm the presence of a living person (3) or in the case of an even stricter validation, then a value of a20.8 can be used.

[0221] Another aspect of the method according to the invention concerns the possibility of using as input map of the neural network (270) the thermal data and when applicable, the visible data and / or the near infrared data, no longer at a single instant noted t0 but during a succession q of successive time instants {t0,...,tq i} with q integer greater than 1, the method therefore being usable from the instant tq4.

[0222] In this context, the dimensions of the neural network (270) are adapted to take into account the increase in the number of elements of the input map: in a preferred embodiment according to the invention, the number i, an integer representing the number of output maps generated by the first convolution layer of the neural network (270), compared to the use of a single instant of capture of the thermal and visual data, noted to, is multiplied by the number of elements q taken into account by the method and in the same way, the input map comprises the q elements I(z)h>v, corresponding to the thermal and visible data and / or near infrared data of the instants {t0,...,tq 1} considered, the other steps of the neural network described previously remaining valid.

[0223] A final aspect of the method according to the invention is the possibility of using the image capture system (2) to additionally detect the movement of the person (3) in the environment (1).

[0224] Again, the method uses as input map of the neural network (270) the thermal data and when applicable, the visible data and / or near infrared data, no longer at a single instant noted t0 but during a succession q of successive time instants {t0,...,tq 1} with q integer greater than 1, the method being accessible from the instant tq4.

[0225] The shooting system (2) is arranged so as to be able to take pictures of the person (3) in the environment (1) and therefore in a preferred embodiment according to the invention, by using the optics (220) and (240) defined previously having the largest field of view.

[0226] Using the neural network (270) according to the invention, the processor (271) of the computer (27) calculates the successive probabilities PR0...PRq 1, from the input maps taken at each instant to,... tq i .

[0227] When successively, u probabilities PR satisfy the acceptance criterion PR cL' u being an integer greater than 1, then the method has validated the presence of the person (3) at various positions of the environment (1).

[0228] In this case, using the thermal and / or visible data and / or near infrared data, the processor (271) of the computer can also estimate a positioning of the person (3) in the environment (1).

[0229] In one embodiment according to the invention, the processor (271) extracts from the thermal data the matrix element of greatest value and assigns to this element an estimate of the position of the person (3).

[0230] In another embodiment according to the invention, the processor (271) subtracts from the thermal data DT(td) of the instant td, d integer greater than 1, the thermal data DT(td 1) of the instant Ui and attributes to the matrix element corresponding to the largest absolute value of the difference DT(td)-DT(td4), an estimate of the position of the person (3) in the medium (1).

[0231] Therefore, the method according to the invention is capable of both validating the presence of a person (3) in the environment (1) and providing an estimate of their positioning and / or movement.

[0232] As is self-evident, and as results from the above, the invention is not limited to the particular embodiments which have just been described; on the contrary, it embraces all variants thereof.

Claims

Claims

1. Image processing method intended for biometric applications, comprising the following steps: acquiring data representative of an image in the thermal infrared spectral domain, called thermal data, of a person in an environment by means of a matrix sensor, and is essentially characterized in that it comprises the following steps: storing this thermal data, transmitting this thermal data to a processor, processing this thermal data by a processor, generating a quantity representative of the probability of an attempted falsification when the thermal data are not representative of the face of a living person placed in front of the matrix sensor but representative of a mask, print or screen placed in front of the matrix sensor for the purpose of falsification, using a neural network whose inputs are the thermal data.

2. Method according to claim 1 characterized in that a step is provided for acquiring the data representative of the person's face to constitute the thermal data.

3. Method according to any one of claims 1 to 2 characterized in that it comprises a step for acquiring data representative of the same person, called visible data, in the visible spectral domain by means of a camera, the neural network additionally having the visible data among its inputs.

4. Method according to claim 3, characterized in that a step is provided for acquiring the data representative of the person's face to constitute the visible data.

5. Method according to any one of claims 3 to 4, characterized in that a step is provided for using data representative of a three-dimensional reconstruction of the person in the environment.

6. Method according to any one of claims 3 to 5, characterized in that a step is provided for acquiring data representative of an image, called near infrared data, of the person in the near infrared spectral domain by means of the same camera, the neural network having in addition the near infrared data among its inputs.

7. Method according to any one of claims 1 to 6, characterized in that a convolutional neural network is used.

8. Method according to claim 7 characterized in that a convolutional neural network is used comprising two convolution layers, two sub-sampling layers and a fully connected layer.

9. Method according to any one of claims 1 to 8, characterized in that a step is provided for acquiring the thermal data over time to constitute the inputs of the neural network.

10. Method according to any one of claims 3 to 9, characterized in that a step is provided for acquiring the visible data over time to constitute the inputs of the neural network.

11. Method according to any one of claims 6 to 10, characterized in that a step is provided for acquiring the near infrared data over time to constitute the inputs of the neural network.

12. Method according to any one of claims 10 to 11, characterized in that a step is provided for generating a quantity representative of the probability of movement of the person.

13. Method according to any one of claims 3 to 12 characterized in that a step is provided for resizing the size of the visible data and / or the near infrared data to the size of the thermal data.

14. Method according to any one of claims 3 to 13 characterized in that a step is provided for introducing a time shift between the thermal data and the visible data.

15. Image processing system intended for biometric applications, comprising: a system for capturing images, arranged so as to enable the capture of images in the thermal infrared range of a person in a medium, the shooting means providing digital data representative of the images called thermal data, and is essentially characterized by: means for storing the thermal data, means for transmitting these thermal data, means of exploitation for exploiting these thermal data, the means of exploitation comprising a neural network arranged to generate a quantity representative of the probability of an attempted falsification when the thermal data are not representative of the face of a living person placed in front of the shooting system but representative of a mask, print or screen placed in front of the shooting system for the purpose of falsification, the inputs of the neural network being the thermal data.

16. System according to claim 15 characterized in that the image capture system is arranged so as to allow the capture of images in the thermal infrared range of the person's face.

17. System according to any one of claims 15 to 16, characterized in that the image capture system is also arranged so as to allow the capture of images of the person in the visible domain, the image capture means providing digital data representative of the images called visible data, the neural network also having the visible data among its inputs.

18. System according to claim 17 characterized in that the operating means are arranged so as to allow the taking of images of the person's face.

19. System according to any one of claims 17 to 18, characterized in that the exploitation means use data representative of a three-dimensional reconstruction of the person in the environment.

20. System according to any one of claims 17 to 19 characterized in that the image capture system is arranged to allow the capture of images of the person in the near infrared range, the means providing digital data of the images called near infrared data, the network of neurons having in addition near infrared data among its inputs.

21. System according to any one of claims 15 to 20 characterized in that the neural network is arranged as a convolutional neural network.

22. System according to claim 21 characterized in that the neural network is arranged as a convolutional neural network comprising two convolution layers, two subsampling layers and a fully connected layer.

23. System according to any one of claims 15 to 22 characterized in that the exploitation means are arranged to use the thermal data over time to constitute the inputs of the neural network.

24. System according to any one of claims 17 to 23 characterized in that the exploitation means are arranged to use the data visible over time as inputs to the neural network.

25. System according to any one of claims 20 to 24 characterized in that the exploitation means are arranged to use the near infrared data over time as inputs to the neural network.

26. System according to any one of claims 15 to 25, characterized in that the exploitation means are arranged to exploit the thermal data over time and generate a quantity representative of the probability of movement of the person over time.

27. ​​System according to any one of claims 17 to 26 characterized in that the operating means are arranged to resize the size of the visible data and / or near infrared data to the size of the thermal data.

28. System according to any one of claims 17 to 27 characterized in that the exploitation means are arranged so as to introduce a time shift between the thermal data and the visible data.

Images