Infrared image processing and living body detection method and device

Abstract

The application discloses an infrared image processing and living body detection method and device, which are used for realizing higher-precision image detection of living bodies or non-living bodies. The processing method comprises the following steps: determining an infrared image to be processed, wherein the infrared image to be processed comprises at least one of a living body infrared image and a non-living body infrared image collected by a target object; performing at least one operation of first enhancement processing and second enhancement processing on the infrared image to be processed, to obtain a target image corresponding to the infrared image to be processed, the target image being used for training a living body detection model, and the living body detection model being used for determining whether a detection object is a living body; the first enhancement processing comprises the following steps: dividing the infrared image to be processed into a plurality of image blocks, and adjusting positions of at least part of the image blocks in the infrared image; the second enhancement processing comprises the following steps: fusing the infrared image to be processed with a preset halo image; and the halo image is obtained by processing an image containing light information.

Description

Technical Field

[0001] This application relates to the field of image technology, and in particular to an infrared image processing, liveness detection method and apparatus. Background Technology

[0002] The following description of the background technology may include insights, discoveries, understandings, or disclosures or associations with at least some examples of embodiments of this application, as well as disclosures provided by this application that are unknown to related prior art. Some of these contributions of this application may be specifically pointed out below, while others will be apparent from the relevant context.

[0003] Facial recognition, as a key application of artificial intelligence technology, has experienced rapid development in recent years. Its various applications are becoming increasingly common in daily life, such as facial recognition payment and access control, bringing significant convenience. However, it also presents numerous security risks, such as the theft of facial information and the potential for unauthorized intrusion. Therefore, security is a crucial factor that facial recognition systems must consider. To defend against attacks using fake faces, liveness detection technology is essential. Liveness detection technology is typically used to defend against attacks such as printed photo attacks, screen attacks, and mask attacks.

[0004] Currently, commonly used liveness detection methods fall into two categories: one requires human cooperation, which determines whether a person is real by observing their blinking, head shaking, mouth opening, and other actions; the other is a non-human-cooperative method, which captures facial images and extracts liveness features from them to determine whether a person is real. The non-cooperative method is seamless, convenient, quick, and does not cause discomfort, making it more suitable for real-world applications. Summary of the Invention

[0005] This application provides an infrared image processing, liveness detection method and apparatus to achieve higher precision in liveness or non-liveness image detection.

[0006] An infrared image processing method provided in this application includes:

[0007] The infrared image to be processed is determined, and the infrared image to be processed includes at least one of a live infrared image and a non-live infrared image acquired from the target object;

[0008] The infrared image to be processed is subjected to at least one of a first enhancement processing and a second enhancement processing to obtain a target image corresponding to the infrared image to be processed. The target image is used to train a liveness detection model, which is used to determine whether the detected object is a live body.

[0009] The first enhancement process includes: dividing the infrared image to be processed into multiple image blocks, and adjusting the position of at least some of the image blocks in the infrared image;

[0010] The second enhancement process includes: fusing the infrared image to be processed with a preset halo image; the halo image is obtained by processing an image containing illumination information.

[0011] This method obtains the target image through the first enhancement process and the second enhancement process, and uses the target image for model training, thereby achieving higher accuracy in detecting live or non-live images.

[0012] In one optional implementation, adjusting the position of at least some of the image blocks in the infrared image includes:

[0013] According to the preset positions on the infrared image to be processed, N1 image blocks are selected from the plurality of image blocks, where N1 is an integer greater than 1;

[0014] According to the preset position adjustment principle, each image block in the N1 image blocks is adjusted to its corresponding target position in the infrared image to be processed; wherein, the target position of each image block in the infrared image to be processed is different from the original position of each image block; the original position is the position of the corresponding image block in the infrared image to be processed before the first enhancement processing is performed on the infrared image to be processed.

[0015] In one optional implementation, fusing the infrared image to be processed with a preset halo image includes:

[0016] The pixel values ​​of at least some pixel locations in the infrared image to be processed are weighted and superimposed with the pixel values ​​of the target pixel locations in the halo image, wherein:

[0017] The target pixel position corresponds to the at least some pixel positions.

[0018] This application provides a liveness detection method, including:

[0019] Acquire the infrared image of the object to be detected;

[0020] The trained liveness detection model is used to detect the infrared image to determine whether the object being detected is a live body; the liveness detection model is trained using the target image.

[0021] In one optional implementation, the liveness detection model includes N2 liveness detection branches and a classification layer, where N2 is an integer greater than 1;

[0022] The step of using a trained liveness detection model to detect the infrared image to be detected and determining whether the detected object is a live object includes:

[0023] Using each of the N2 liveness detection branches, the infrared image to be detected is detected respectively, and the sub-detection results corresponding to each liveness detection branch are obtained.

[0024] The classification layer is used to fuse the obtained sub-detection results to determine whether the detected object is a live object; wherein, the N2 liveness detection branches are trained using target images of different categories.

[0025] Therefore, the embodiments of this application can enhance the anti-spoofing capability against various types of attacks on liveness detection, thereby further improving the detection accuracy of liveness or non-liveness.

[0026] In one alternative implementation, the liveness detection model is trained in the following manner:

[0027] Obtain N2 sets of target images; wherein, different sets of target images contain target images of different categories, and the target images contain annotation information on whether the target object is a living organism;

[0028] A liveness detection branch is trained using each of the N2 target image sets;

[0029] Using the trained liveness detection branches, at least a portion of the target images contained in the N2 target image sets are detected to obtain the sub-detection results corresponding to each liveness detection branch. The classification layer is then used to fuse the obtained sub-detection results to determine whether the target objects contained in the at least a portion of the target images are live. Based on the detection results and the corresponding annotation information, the network parameters of the classification layer are adjusted to obtain the trained liveness detection model.

[0030] In one alternative implementation, the different categories of target images include at least different categories of inanimate infrared images.

[0031] An image processing apparatus provided in this application includes:

[0032] Memory, used to store program instructions;

[0033] A processor is configured to invoke program instructions stored in the memory and execute any of the methods described according to the obtained program.

[0034] Another embodiment of this application provides a computing device including a memory and a processor, wherein the memory is used to store program instructions, and the processor is used to call the program instructions stored in the memory and execute any of the methods described above according to the obtained program.

[0035] Furthermore, according to embodiments, for example, a computer program product for a computer is provided, which includes software code portions that, when the product is run on the computer, perform the steps of the methods defined above. The computer program product may include a computer-readable medium on which the software code portions are stored. Furthermore, the computer program product may be directly loaded into the computer's internal memory and / or sent via a network through at least one of an upload process, a download process, and a push process.

[0036] Another embodiment of this application provides a computer-readable storage medium storing computer-executable instructions for causing the computer to perform any of the methods described above. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A schematic flowchart illustrating the model training method provided in this application embodiment;

[0039] Figure 2 A schematic flowchart illustrating the face detection method provided in this application embodiment;

[0040] Figure 3 This is a schematic diagram of face image data reconstruction provided in an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of face image halo overlay provided in an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the structure of a single convolutional neural network provided in an embodiment of this application;

[0043] Figure 6 This is a schematic diagram of the structure of the fused convolutional neural network provided in the embodiments of this application;

[0044] Figure 7 This is a schematic diagram of the structure of a fully connected layer in a fused convolutional neural network provided in an embodiment of this application;

[0045] Figure 8 A flowchart illustrating an infrared image processing method provided in an embodiment of this application;

[0046] Figure 9 A schematic flowchart of a liveness detection method provided in an embodiment of this application;

[0047] Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0048] Figure 11 This is a schematic diagram of the structure of a liveness detection device provided in an embodiment of this application;

[0049] Figure 12 This is a schematic diagram of the structure of an infrared image processing device provided in an embodiment of this application;

[0050] Figure 13 This is a schematic diagram of the structure of a liveness detection device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0052] Definitions:

[0053] Non-living face image: When infrared image acquisition is performed on a target object, the target object is not a real living person, but an image, mask, or other type of object. However, a face image can be obtained by acquiring an infrared image of the object. The acquired face image is called a non-living face image. Conversely, if the target object is a real living person, the face image acquired from the target object is called a living face image.

[0054] Halo image: The halo image is an image obtained by processing an image containing illumination information.

[0055] This application provides an infrared image processing, liveness detection method and apparatus to achieve higher precision in liveness or non-liveness image detection.

[0056] The method and apparatus are based on the same concept of the application. Since the methods and apparatus solve problems in similar ways, the implementation of the apparatus and methods can refer to each other, and the repeated parts will not be described again.

[0057] The terms "first," "second," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] The following examples and embodiments are to be understood as illustrative only. While this specification may refer to "a," "an," or "some" examples or embodiments in several places, this does not mean that every such reference relates to the same example or embodiment, nor does it mean that the feature applies only to a single example or embodiment. Individual features of different embodiments may also be combined to provide other embodiments. Furthermore, terms such as "comprising" and "including" should be understood not to limit the described embodiments to consisting only of those features mentioned; such examples and embodiments may also include features, structures, units, modules, etc., not specifically mentioned.

[0059] The various embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be noted that the order in which the embodiments are presented in this application represents only a chronological order and does not represent the superiority or inferiority of the technical solutions provided by the embodiments.

[0060] The technical solutions provided in this application involve how to avoid high-cost data annotation on images, how to extract effective live features from images, how to consider the impact of lighting in real-world scenarios on face detection and judgment, and how to defend against various types of attacks on face recognition systems.

[0061] The technical solution provided in this application does not require pixel-level annotation on the image, resulting in lower costs. It uses a convolutional neural network to perform liveness detection on the image, employing a data reconstruction enhancement method in the preprocessing stage to effectively extract liveness features and reduce the interference of content features on the image. Furthermore, this application uses a halo overlay method to simulate real-world lighting conditions, enhancing the robustness of the face detection model to lighting conditions. The convolutional neural network used in this application is a fusion of multiple convolutional neural networks, each with good discrimination capabilities against specific attack types. After fusion, the entire network can distinguish various attack methods, achieving high-precision liveness detection.

[0062] The technical solutions provided in this application are illustrated using face detection as an example, but the technical solutions provided in this application are also applicable to the detection of other objects.

[0063] The following is combined with Figure 1 and Figure 2 This application provides a detailed description of the technical solutions provided in its embodiments.

[0064] The entire training process of the convolutional neural network used for face detection provided in this application embodiment is as follows: Figure 1 As shown, it includes:

[0065] S101. Acquire infrared training images of the target object. Specifically, this includes:

[0066] The target object is, for example, a person. Of course, different types of target objects can be set according to actual needs.

[0067] For example, by acquiring images of human faces using an infrared camera, various types of infrared liveness training data can be obtained; and / or,

[0068] Images of fake faces (i.e., non-living faces) are captured by infrared cameras on printed images, masks, screen images, etc., thereby obtaining non-living training data.

[0069] S102. Extract face images from the infrared training images using face detection;

[0070] S103. Preprocess the face image.

[0071] The image obtained after the preprocessing is referred to as the target image in this embodiment of the application.

[0072] Optionally, each target image includes annotation information indicating whether the target object is a living organism. That is, the images used for training the model in this embodiment are labeled images. Of course, the technical solutions provided in this embodiment are also applicable to unlabeled images.

[0073] The preprocessing includes, for example, data reconstruction and / or halo overlay.

[0074] It should be noted that, Figure 1 The process includes data reconstruction and halo overlay, and data reconstruction is performed first and then halo overlay. However, the embodiments of this application are not limited to this method, but are merely illustrative. Only one of the preprocessing methods can be used. If both data reconstruction and halo overlay are used, the specific order is not limited. Halo overlay can be performed first and then data reconstruction can be performed.

[0075] The data recombination can also be referred to as the first enhancement process; correspondingly, the halo overlay can also be referred to as the second enhancement process.

[0076] S104. Using different categories of preprocessed face images, train multiple convolutional neural networks.

[0077] The multiple convolutional neural networks are, in effect, multiple liveness detection branches. The target image set used for training each liveness detection branch includes multiple target images. Different liveness detection branches correspond to different target image sets, i.e., they correspond to different types of target images.

[0078] For example, three convolutional neural networks can be trained (of course, the embodiments of this application are not limited to three, and can be one or any number of them).

[0079] The training of three convolutional neural networks includes, for example:

[0080] To address non-living attacks related to printing, a first convolutional neural network is trained using non-living face image data from printed sources. This non-living face image data from printed sources specifically includes, for example, printed image data such as color photographs and pictures.

[0081] To address mask-based non-living attacks, a second convolutional neural network is trained using mask-based non-living face image data.

[0082] A third convolutional neural network is trained using facial image data outside the categories mentioned above. For example, for attack types that are difficult to defend against, such as attacks in darkroom scenes, attacks in strong backlight scenes, and infrared printing attacks, non-living facial image data collected in any one or more of these scenarios, such as facial images collected in darkroom scenes, facial images collected in strong backlight scenes, and infrared printed images, are used to train the third convolutional neural network. The infrared printed images mentioned above, unlike the aforementioned non-living facial images used in printing, are black and white images.

[0083] S105. The three convolutional neural networks are fused into one convolutional neural network to obtain a fused convolutional neural network.

[0084] In other words, in this embodiment of the application, as an optional implementation, each liveness detection branch trained is used to detect at least a portion of the target images contained in the target image set, and sub-detection results corresponding to each liveness detection branch are obtained. The sub-detection results are then fused using a classification layer (or a fully connected layer, i.e., an fc layer) to determine whether the target object contained in the at least a portion of the target images is a live object. Based on the detection results and the corresponding annotation information, the network parameters of the classification layer are adjusted to obtain the trained liveness detection model, i.e., a fused convolutional neural network.

[0085] After obtaining the fused convolutional neural network, in practical applications, see [link to relevant documentation]. Figure 2 The face detection method provided in this application includes:

[0086] S201. Acquire the infrared image of the object to be detected;

[0087] S202. Perform face detection on the infrared image of the detected object to obtain an infrared face image;

[0088] S203. Use the fused convolutional neural network to detect the infrared face image, thereby determining whether the detected object is a live or non-live body;

[0089] S204. Output the detection result of whether the detected object is a living or non-living body.

[0090] The following examples illustrate the data recombination, halo overlay, training of convolutional neural networks, and fusion mentioned above.

[0091] (1) Data reconstruction of facial images:

[0092] The data recombination method used in the embodiments of this application is mainly used to improve the ability to extract facial activity features and reduce the impact of content features on liveness detection.

[0093] The facial activity features refer to facial features used to distinguish whether the detected object is a living or non-living person; the content features are features used to distinguish the identity or attributes of the detected object, including facial structural features, etc.

[0094] See Figure 3 (To protect privacy, the face area has been pixelated; in practical applications, a clear image can be used.) This is achieved by analyzing the detected infrared face image. Figure 3The leftmost image is divided into nine regions, each labeled with a number. The positions of regions 4, 5, 6, and 8 in the original image are randomly shuffled. These nine shuffled image regions are then combined to form a new face image. Figure 3 The rightmost image is shown.

[0095] It should be noted that, as an optional implementation, considering that image blocks 4, 5, 6, and 8 are the main regions of the face, containing facial features and rich content features, the positions of these four regions in the original image are shuffled in this embodiment. This can weaken the content features of the face, allowing the subsequent training of the convolutional neural network to focus more on the activity features of the face, thereby improving the robustness of the face detection model.

[0096] (2) Regarding halo overlay on facial images:

[0097] The halo overlay method used in the embodiments of this application is mainly used to improve the robustness of the face detection model to lighting in real-world scenarios and enhance the effect of liveness detection in practical applications.

[0098] See Figure 4 First, in a real-world scenario, infrared cameras capture images of illumination. These images contain light spots and halos. Images of these light spots and halos are then extracted to obtain halo images, such as... Figure 4 The two middle images are shown.

[0099] Then, randomly select some face images from the training data, for example... Figure 4 The leftmost image shows a face image randomly overlaid with one of various halo images (halo images with light spots, halo images with apertures, etc.) to protect privacy. The resulting face image after overlaying the halo image (e.g.) Figure 4 The image shown on the far right is used to train a convolutional neural network. The halo image can also be referred to as a natural light image.

[0100] One method for superimposing a face image onto a halo image is linear weighted superposition. For any pixel location, assuming the pixel value at any location in the face image is `face`, and the pixel value at the same location in the halo image is `light`, with a weight of `a` (e.g., a random number between 0.5 and 1), the pixel value at the same location in the superimposed image is `face*a + light*(1-a)`. This method simulates various lighting conditions in real-world scenarios, making the model robust to lighting conditions and improving liveness detection performance in practical applications.

[0101] (3) Regarding the training of a single convolutional neural network and the fusion of multiple convolutional neural networks:

[0102] The structure of each convolutional neural network used in the embodiments of this application is as follows: Figure 5 As shown, the model includes multiple convolutional layers (conv), pooling layers (pool), and finally two fully connected layers (fc). Optimization training is then performed using a softmax loss function. Convolutional layers are used for convolution operations, and pooling layers are used for downsampling. Each neuron in a fully connected layer is fully connected to all neurons in the layer preceding it. Fully connected layers can integrate local information with class discriminative power from convolutional or pooling layers. Softmax loss is a commonly used loss function in convolutional neural networks. The loss function is used to evaluate the difference between the predicted and actual values ​​of the face detection model. The optimization training, i.e., supervised training of the convolutional neural network using the loss function, determines the weight parameters in the face detection model. The face detection model described in this embodiment is the convolutional neural network used.

[0103] In this embodiment, the data used for training each convolutional neural network is different, and the types of non-living attacks it defends against are also different. This embodiment uses three convolutional neural networks, such as... Figure 1 As shown, the training data for the first convolutional neural network mainly consists of non-human attacks such as printing attacks; the training data for the second convolutional neural network mainly consists of non-human attacks such as mask attacks; and the training data for the third convolutional neural network mainly consists of attack types that are more difficult to defend against, such as attacks in dark rooms, attacks in strong backlight scenes, and infrared printing attacks. Since screen-based attacks (such as displaying faces on mobile phones or tablets) do not image under infrared cameras, screen-based data is not required. After the three models are trained, they can respectively defend against printing attacks, mask attacks, and attacks in difficult scenes.

[0104] A convolutional neural network (CNN) includes an input layer, hidden layers, and an output layer. The hidden layers include convolutional layers, pooling layers, and fully connected layers; the softmax layer is the output layer, typically connecting to the last fully connected layer in the hidden layers. The softmax layer is used in multi-class classification to map the outputs of multiple neurons to the 0 and 1 intervals, thereby performing multi-class classification. This application's embodiment uses a fused convolutional neural network method, merging the above three convolutional neural networks into a single fused convolutional neural network. The structure of the resulting fused convolutional neural network is as follows... Figure 6 As shown, the infrared face image is input into a fusion convolutional neural network (CNN), which simultaneously inputs it into three CNNs. The last fully connected layer of each CNN is connected to a softmax layer. These three softmax layers are then connected to a concat layer, followed by a fully connected (FC) layer, and finally, a softmax layer outputs the classification result. The concat layer concatenates two or more feature images along the channel or number dimension. Here, the concat layer connects the outputs of the three softmax layers. For example, if the three softmax layers output [0.2, 0.8], [0.6, 0.4], and [0.5, 0.5] respectively, then [0.2, 0.8] indicates that the probability of identifying a live subject is 0.2 and the probability of identifying a non-live subject is 0.8 in the recognition result output by the first CNN. Therefore, after passing through the concat layer, the output result is [0.2, 0.8, 0.6, 0.4, 0.5, 0.5]. Figure 6 The fully connected (FC) layer shown has two nodes, used for binary classification. That is to say, Figure 6 The concat layer connects the outputs of three softmax layers, resulting in a result with six nodes (e.g., [0.2, 0.8, 0.6, 0.4, 0.5, 0.5]). These six nodes are then paired with two nodes from the fully connected (FC) layer, resulting in two output nodes, ultimately yielding two values ​​representing the probabilities of a live and a non-live individual. Specifically:

[0105] The embodiments of this application provide the fused convolutional neural network's fc layer (i.e., ... Figure 6 The example structure of fc shown is as follows Figure 7 As shown, x1 to x6 are the upper-level outputs, such as the six nodes [0.2, 0.8, 0.6, 0.4, 0.5, 0.5] mentioned above, with the parameter w pre-set. ij , representing x among the six nodes i and the two nodes a jThe weight parameters between the nodes are pre-set as bias parameters b1 and b2, where i takes values ​​from 1 to 6 and j takes values ​​from 1 to 2. Then, the fc layer outputs the values ​​a1 and a2 of the two nodes according to the following two formulas (of course, this embodiment is not limited to these formulas; the setting and selection of parameters can be determined according to actual needs):

[0106] a1 = w 11 *x1+w 21 *x2+w 31 *x3+w 41 *x4+w 51 *x5+w 61 *x6+b1

[0107] a2 = w 12 *x1+w 22 *x2+w 32 *x3+w 42 *x4+w 52 *x5+x 62 x6+b2

[0108] After training, the fc layer is w ij With b1 and b2 fixed, the values ​​of the last two nodes can be obtained from the above calculations.

[0109] Figure 6 The structure shown is viewed as a fused convolutional neural network. A single model combines the recognition capabilities of three individual models, and a single input shares the preprocessing resources of all three models. This reduces preprocessing time in practical applications.

[0110] The model parameters of the three convolutional neural networks are trained separately. After merging, the model parameters of each individual convolutional neural network remain unchanged and only need to be trained separately. Figure 6 The parameters of the fully connected (FC) layer, such as the weight parameter w mentioned above. ij With bias parameters b1 and b2, the fused convolutional neural network, after training, can determine whether an individual is alive or not in infrared face images. This fusion method combines the functions of three convolutional neural networks with different defense types, resulting in a fused convolutional neural network capable of defending against various attack types.

[0111] In summary, the technical solution provided in this application does not require pixel-level data annotation, resulting in low cost. The data recombination method effectively extracts liveness features and reduces the interference of content features on liveness detection. The halo overlay method makes the model more robust to real-world lighting conditions. Furthermore, the method using fused convolutional neural networks provides excellent anti-spoofing capabilities against various types of attacks, achieving high-precision liveness detection.

[0112] The methods provided in the embodiments of this application are summarized below:

[0113] See Figure 8 This application provides an infrared image processing method, comprising:

[0114] S801. Determine the infrared image to be processed, wherein the infrared image to be processed includes at least one of a live infrared image and a non-live infrared image acquired from the target object;

[0115] S802. Perform at least one of a first enhancement processing and a second enhancement processing on the infrared image to be processed to obtain a target image corresponding to the infrared image to be processed. The target image is used to train a liveness detection model, and the liveness detection model is used to determine whether the detected object is a live body; wherein:

[0116] The first enhancement process (e.g., the data reconstruction) includes: dividing the infrared image to be processed into multiple image blocks, and adjusting the position of at least some of the multiple image blocks in the infrared image;

[0117] The second enhancement process (e.g., the halo overlay) includes: fusing the infrared image to be processed with a preset halo image; the halo image is obtained by processing an image containing illumination information.

[0118] In this embodiment, the use of data recombination effectively extracts liveness features and reduces the interference of content features on liveness detection. Furthermore, it eliminates the need for pixel-level data annotation, resulting in low cost.

[0119] The halo overlay method used in this embodiment makes the trained convolutional neural network more robust to lighting conditions in real-world scenarios.

[0120] In one optional implementation, adjusting the position of at least some of the image blocks in the infrared image includes:

[0121] According to a preset position on the infrared image to be processed, such as the position of facial features, N1 image blocks are selected from the plurality of image blocks, where N1 is an integer greater than 1; for example, in the above embodiment, N1 equals 4.

[0122] According to the preset position adjustment principle, each image block in the N1 image blocks is adjusted to its corresponding target position in the infrared image to be processed; wherein, the target position of each image block in the infrared image to be processed is different from the original position of each image block; the original position is the position of the corresponding image block in the infrared image to be processed before the first enhancement processing is performed on the infrared image to be processed.

[0123] The preset position adjustment principle can be, for example, a random position adjustment, an adjustment according to a pre-set position, or a symmetrical swap according to one or more reference lines. The reference lines can be horizontal lines, vertical lines, diagonal lines, or even arbitrary lines; specific implementations in this application are not limited.

[0124] In one optional implementation, fusing the infrared image to be processed with a preset halo image includes:

[0125] The pixel values ​​of at least some pixel locations in the infrared image to be processed are weighted and superimposed with the pixel values ​​of the target pixel locations in the halo image, wherein:

[0126] The target pixel position corresponds to the at least some pixel positions.

[0127] See Figure 9 This application provides a liveness detection method, comprising:

[0128] S901. Obtain the infrared image of the object to be detected;

[0129] For example, the detection object can be a human body image, a face image extracted from a human body image, or a directly captured face image.

[0130] S902. The infrared image to be detected is detected using the trained liveness detection model to determine whether the detected object is a live body. The liveness detection model is trained using the target image, or it can be said that a fused convolutional neural network trained by the above method is used for liveness detection.

[0131] In one optional implementation, the liveness detection model includes N2 liveness detection branches and a classification layer, where N2 is an integer greater than 1; for example, in the above implementation, N2 equals 3.

[0132] The step of using a trained liveness detection model to detect the infrared image to be detected and determining whether the detected object is a live object includes:

[0133] Using each of the N2 liveness detection branches, the infrared image to be detected is detected respectively, and the sub-detection results corresponding to each liveness detection branch are obtained.

[0134] The classification layer is used to fuse the obtained sub-detection results to determine whether the detected object is a live object; wherein, the N2 liveness detection branches are trained using target images of different categories.

[0135] Therefore, the embodiments of this application can enhance the anti-spoofing capability against various types of attacks on liveness detection, thereby further improving the detection accuracy of liveness or non-liveness.

[0136] In one alternative implementation, the liveness detection model is trained in the following manner:

[0137] Obtain N2 sets of target images; wherein, different sets of target images contain target images of different categories, and the target images contain annotation information on whether the target object is a living organism;

[0138] A liveness detection branch is trained using each of the N2 target image sets;

[0139] Using the trained liveness detection branches, at least a portion of the target images contained in the N2 target image sets are detected to obtain the sub-detection results corresponding to each liveness detection branch. The classification layer is then used to fuse the obtained sub-detection results to determine whether the target objects contained in the at least a portion of the target images are live. Based on the detection results and the corresponding annotation information, the network parameters of the classification layer are adjusted to obtain the trained liveness detection model (i.e., the fused convolutional neural network).

[0140] The embodiments of this application, by using a fused convolutional neural network, can have good anti-spoofing capabilities against various types of attacks, and can achieve higher accuracy in detecting live or non-live faces.

[0141] For example, the first convolutional neural network, the second convolutional neural network, and the third convolutional neural network respectively obtain the first detection result, the second detection result, and the third detection result;

[0142] The first detection result, the second detection result, and the third detection result are each processed by an output layer and then output to a connection layer; for example, combined with Figure 6 The first detection result, the second detection result, and the third detection result are respectively output to different softmax layers;

[0143] The result processed by the connection layer is output to a fully connected layer; for example, combined with Figure 6 The outputs of the three softmax layers are processed by the concat layer and then output to the fully connected layer.

[0144] The result processed by the fully connected layer is finally processed by an output layer to output the detection result of whether the detected object is alive or not. For example, combined with Figure 6 The result processed by the FC layer is output to the last softmax layer, which then outputs the final result.

[0145] In addition to the methods described in the above embodiments, the three branches (i.e., the first convolutional neural network, the second convolutional neural network, and the third convolutional neural network) output the result and confidence level of whether the image is a certain type of non-living image, respectively. The classification layer can also select the result with the highest confidence level to output to the last softmax layer, which outputs the final result.

[0146] In one alternative implementation, the different categories of target images include at least different categories of inanimate infrared images.

[0147] See Figure 10 An image processing apparatus provided in this application includes:

[0148] Memory 11 is used to store program instructions;

[0149] The processor 12 is used to call the program instructions stored in the memory and execute any of the methods described in accordance with the obtained program. These methods can be either the infrared image processing method or the liveness detection method. The specific details will not be elaborated further.

[0150] Specifically, for example, see Figure 11 This application provides a liveness detection device, comprising:

[0151] Processor 600 is used to read the program from memory 620 and execute the following procedures:

[0152] Acquire the infrared image of the object to be detected;

[0153] The trained liveness detection model, such as the fused convolutional neural network, is used to detect the infrared image to be detected and determine whether the detected object is a live body.

[0154] Transceiver 610 (optional) is used to receive and transmit data under the control of processor 600. For example, the transceiver can transmit any type of data in a wired or wireless form, such as for acquiring an infrared image of a target object.

[0155] Among them, Figure 11 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 600 and memory represented by memory 620 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 610 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. For different user equipment, the user interface 630 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0156] The processor 600 is responsible for managing the bus architecture and general processing, while the memory 620 can store the data used by the processor 600 when performing operations.

[0157] Optionally, the processor 600 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a CPLD (Complex Programmable Logic Device).

[0158] See Figure 12 Another infrared image processing apparatus provided in this application includes:

[0159] The infrared image to be processed unit 121 is used to determine the infrared image to be processed, wherein the infrared image to be processed includes at least one of a live infrared image and a non-live infrared image acquired from the target object;

[0160] Enhancement processing unit 122 is configured to perform at least one of a first enhancement processing and a second enhancement processing on the infrared image to be processed to obtain a target image corresponding to the infrared image to be processed. The target image is used to train a liveness detection model, and the liveness detection model is used to determine whether the detected object is a live body.

[0161] The first enhancement process includes: dividing the infrared image to be processed into multiple image blocks, and adjusting the position of at least some of the image blocks in the infrared image;

[0162] The second enhancement process includes: fusing the infrared image to be processed with a preset halo image; the halo image is obtained by processing an image containing illumination information.

[0163] Other specific implementation methods that are the same as or corresponding to the above-described infrared image processing method will not be repeated here.

[0164] See Figure 13 Another liveness detection device provided in this application includes:

[0165] The infrared image acquisition unit 131 is used to acquire the infrared image of the object to be detected.

[0166] The detection unit 132 is used to detect the infrared image to be detected using a trained liveness detection model to determine whether the detected object is a live body; the liveness detection model is trained using the target image.

[0167] Other specific implementation methods that are the same as or corresponding to the above-described live detection method will not be repeated here.

[0168] It should be noted that the division of units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0169] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0170] This application provides a computing device, which may specifically be a desktop computer, portable computer, smartphone, tablet computer, personal digital assistant (PDA), etc. The computing device may include a central processing unit (CPU), memory, input / output devices, etc. Input devices may include a keyboard, mouse, touchscreen, etc., and output devices may include display devices, such as a liquid crystal display (LCD) or a cathode ray tube (CRT).

[0171] The memory may include read-only memory (ROM) and random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In the embodiments of this application, the memory may be used to store the program of any of the methods provided in the embodiments of this application.

[0172] The processor executes any of the methods described in the embodiments of this application according to the program instructions stored in the memory.

[0173] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the methods described in the above embodiments. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0174] This application provides a computer-readable storage medium for storing computer program instructions used in the apparatus provided in the above-described embodiments, including a program for performing any of the methods provided in the above-described embodiments. The computer-readable storage medium may be a non-transitory computer-readable medium.

[0175] The computer-readable storage medium can be any available medium or data storage device that a computer can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0176] It should be understood that:

[0177] The access technology used by entities in a communication network to transmit traffic can be any suitable current or future technology, such as WLAN (Wireless Local Access Network), WiMAX (Microwave Access Global Interoperability), LTE, LTE-A, 5G, Bluetooth, infrared, etc.; in addition, embodiments may also apply wired technologies, such as IP-based access technologies, such as wired networks or fixed lines.

[0178] An embodiment suitable for implementation as software code or as part thereof and for operation using a processor or processing function is independent of the software code and can be specified using any known or future-developed programming language, such as high-level programming languages ​​such as Objective-C, C, C++, C#, Java, Python, Javascript, other scripting languages, etc., or low-level programming languages ​​such as machine language or assembler.

[0179] The implementation of the embodiments is hardware-independent and can be implemented using any known or future-developed hardware technology or any combination thereof, such as microprocessors or CPUs (central processing units), MOS (metal-oxide-semiconductor), CMOS (complementary MOS), BiMOS (bipolar MOS), BiCMOS (bipolar CMOS), ECL (emitter-coupled logic), and / or TTL (transistor-transistor logic).

[0180] The embodiments may be implemented as individual devices, apparatuses, units, components or functions, or in a distributed manner. For example, one or more processors or processing functions may be used or shared in the process, or one or more processing segments or processing portions may be used and shared in the process, wherein one or more physical processors may be used to implement one or more processing portions dedicated to a particular process as described.

[0181] The device can be implemented by a semiconductor chip, a chipset, or a (hardware) module that includes such a chip or chipset.

[0182] The implementation can also be implemented as any combination of hardware and software, such as ASIC (Application-Specific IC (Integrated Circuit)) components, FPGA (Field Programmable Gate Array) or CPLD (Complex Programmable Logic Device) components or DSP (Digital Signal Processor) components.

[0183] The embodiments can also be implemented as computer program products, including a computer-usable medium in which computer-readable program code is embodied, the computer-usable medium being adapted to perform the processes described in the embodiments, wherein the computer-usable medium may be a non-transitory medium.

[0184] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0185] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0186] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0187] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0188] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An infrared image processing method, characterized by, The method includes: The infrared image to be processed is determined, wherein the infrared image to be processed includes at least one of a live infrared image and a non-live infrared image acquired for the target object; The target object image is determined from the infrared image to be processed, and a target image for training a liveness detection model is generated by performing at least one of a first enhancement processing and a second enhancement processing on the target object image. The liveness detection model includes at least one category of convolutional neural networks for determining whether the detected object is a live object. Different convolutional neural networks correspond to different types of target image sets. The different types include non-live target images of printed objects, non-live target images of masks, and non-live target images other than the printed non-live target images and mask non-live target images. The first enhancement process includes: dividing the infrared image to be processed into multiple image blocks, and adjusting the position of at least some of the image blocks in the infrared image; The second enhancement process includes: fusing the infrared image to be processed with a preset halo image; the halo image is obtained by processing an image containing illumination information; The convolutional neural networks of at least one category used to determine whether a detected object is alive are fused together to obtain a trained liveness detection model.

2. The method of claim 1, wherein, Adjusting the position of at least some of the image blocks in the infrared image includes: According to the preset positions on the infrared image to be processed, N1 image blocks are selected from the plurality of image blocks, where N1 is an integer greater than 1; According to the preset position adjustment principle, each image block in the N1 image blocks is adjusted to its corresponding target position in the infrared image to be processed; wherein, the target position of each image block in the infrared image to be processed is different from the original position of each image block; the original position is the position of the corresponding image block in the infrared image to be processed before the first enhancement processing is performed on the infrared image to be processed.

3. The method according to claim 1 or 2, characterized in that, The step of fusing the infrared image to be processed with a preset halo image includes: The pixel values ​​of at least some pixel locations in the infrared image to be processed are weighted and superimposed with the pixel values ​​of the target pixel locations in the halo image, wherein: The target pixel position corresponds to the at least some pixel positions.

4. A method of detecting living matter, characterized by, The method includes: Acquire the infrared image of the object to be detected; The infrared image to be detected is used to detect whether the object being detected is a living person by using a trained liveness detection model; the liveness detection model is trained using the target image obtained by any one of claims 1-3.

5. The method of claim 4, wherein, The liveness detection model includes N2 liveness detection branches and a classification layer, where N2 is an integer greater than 1; The step of using a trained liveness detection model to detect the infrared image to be detected and determining whether the detected object is a live object includes: Using each of the N2 liveness detection branches, the infrared image to be detected is detected respectively, and the sub-detection results corresponding to each liveness detection branch are obtained. The classification layer is used to fuse the obtained sub-detection results to determine whether the detected object is a live object; wherein the N2 liveness detection branches are trained using target images of different categories.

6. The method of claim 5, wherein, The liveness detection model was trained in the following manner: Obtain N2 sets of target images; wherein, different sets of target images contain target images of different categories, and the target images contain annotation information on whether the target object is a living organism; A liveness detection branch is trained using each of the N2 target image sets; Using the trained liveness detection branches, at least a portion of the target images contained in the N2 target image sets are detected to obtain the sub-detection results corresponding to each liveness detection branch. The classification layer is then used to fuse the obtained sub-detection results to determine whether the target objects contained in the at least a portion of the target images are live. Based on the detection results and the corresponding annotation information, the network parameters of the classification layer are adjusted to obtain the trained liveness detection model.

7. The method according to claim 6, characterized in that, The target images of different categories include at least different categories of in vivo infrared images.

8. An image processing apparatus characterized by comprising: include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method described in any one of claims 1-3 or 4-7 according to the obtained program.

9. A computer program product for a computer, comprising a software code portion that, when the product is run on the computer, is configured to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing the computer to perform the method according to any one of claims 1-3 or 4-7.

Images