Multi-modal human face living body detection method and device and electronic equipment

Abstract

The invention provides a multi-modal face living body detection method and device and electronic equipment, and relates to the technical field of face recognition, and the method comprises the steps: obtaining a to-be-detected face image of a target object; inputting a to-be-detected face image into the face living body detection model to obtain a detection result; wherein the face living body detection model comprises a first processing module used for extracting feature information from a visible light face image, and a second processing module used for extracting feature information from an infrared light face image; the human face living body detection model further comprises a multi-head classification module; and the multi-head classification module is used for carrying out classification prediction on the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result. The multi-modal face living body detection method and device and the electronic equipment provided by the invention are used for improving the robustness aiming at different illumination conditions and different attack types and improving the accuracy of face living body detection.

Description

Technical Field

[0001] This application relates to the field of face recognition technology, and in particular to a multimodal face liveness detection method, device and electronic equipment. Background Technology

[0002] With the development of information technology, facial recognition has been widely used in various fields. In some important identity authentication scenarios, facial recognition also requires liveness detection to ensure that the captured face is a real living person, rather than a photo, video, or other fake image.

[0003] In related technologies, multimodal feature recognition methods can be used for face liveness detection. However, most of the recognition methods in these technologies simply splice the extracted multimodal features, which cannot effectively integrate the features between different modalities. Furthermore, they do not take into account the relationship between global and local facial features, resulting in poor recognition performance.

[0004] Therefore, there is an urgent need for a face liveness detection method to improve robustness under different lighting conditions and different attack types, while also improving the accuracy of face liveness detection. Summary of the Invention

[0005] The purpose of this application is to provide a multimodal face liveness detection method, apparatus, and electronic device to improve robustness under different lighting conditions and different attack types, as well as to improve the accuracy of face liveness detection.

[0006] This application provides a multimodal face liveness detection method, including: A face image of a target object to be detected is acquired; the face image to be detected includes: a visible light face image and an infrared light face image; the face image to be detected is input into a face liveness detection model to obtain a detection result; wherein, the face liveness detection model includes: a first processing module for extracting feature information from the visible light face image, and a second processing module for extracting feature information from the infrared light face image; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face.

[0007] Optionally, the multi-head classification module includes: a first classification head for classifying the feature information extracted by the first processing module, and a second classification head for classifying the feature information extracted by the second processing module; the step of inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: inputting the face image to be detected, a first block image corresponding to a first face image, and a second block image corresponding to a second face image into the face liveness detection model, and generating the detection result based on the classification result of the first classification head and the classification result of the second classification head; wherein, the first face image is a visible light image; the second face image is an infrared light image; the first block image is an image obtained by cropping the first face image; the second block image is an image obtained by cropping the second face image; the classification result of the first classification head is obtained based on the feature information extracted by the first processing module; and the classification result of the second classification head is obtained based on the feature information extracted by the second processing module.

[0008] Optionally, the step of inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: inputting the first face image and the first block image into the first processing module to obtain a first global feature vector for characterizing the global features of the visible light image and a first block feature vector for characterizing the block features of the visible light image; and inputting the second face image and the second block image into the second processing module to obtain a second global feature vector for characterizing the global features of the infrared light image and a second block feature vector for characterizing the block features of the infrared light image.

[0009] Optionally, the face classification result includes: a first classification result, a second classification result, a third classification result, a fourth classification result, a fifth classification result, and a sixth classification result; the step of inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: inputting the first global feature vector into the global feature classification head in the first classification head to obtain the first classification result; and inputting the first block feature vector into the block feature classification head in the first classification head to obtain the second classification result; and inputting the second global feature vector into the global feature classification head in the second classification head to obtain the third classification result; and inputting the second block feature vector into the block feature classification head in the second classification head. In the head, the fourth classification result is obtained; and the first global feature vector and the second global feature vector are input into the attention feature collaborative complementary fusion module to obtain the first collaborative feature expression corresponding to the first global feature vector and the second collaborative feature expression corresponding to the second global feature vector, and the first collaborative feature expression and the second collaborative feature expression are respectively input into the fusion feature classification head to obtain the fifth classification result and the sixth classification result; the step of inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: generating the detection result based on the first classification result, the second classification result, the third classification result, the fourth classification result, the fifth classification result and the sixth classification result.

[0010] Optionally, the step of inputting the first global feature vector and the second global feature vector into the attention feature collaborative complementary fusion module to obtain the first collaborative feature expression corresponding to the first global feature vector and the second collaborative feature expression corresponding to the second global feature vector includes: adjusting the dimensions of the first global feature vector and the second global feature vector; inputting the adjusted first global feature vector into the corresponding linear layer to obtain a first query feature vector, a first key feature vector, and a first value feature vector; and inputting the adjusted second global feature vector into the corresponding linear layer to obtain a second query feature vector, a second key feature vector, and a second value feature vector; performing matrix multiplication of the first key feature vector and the second query feature vector to obtain a first result, and inputting the first result into a normalized exponential function for processing to obtain a first cross-attention matrix; and multiplying the first key feature vector with the first key feature vector and the second query feature vector... The cross-attention matrix is ​​multiplied to obtain the first cross-attention output result; the second key feature vector and the first query feature vector are multiplied by a matrix to obtain the second result, and the second result is input into a normalized exponential function for processing to obtain the second cross-attention matrix. The second value feature vector is multiplied by the second cross-attention matrix to obtain the second cross-attention output result; the first cross-attention output result is input into the corresponding mapping linear layer to obtain the first mapping feature vector, and the first mapping feature vector is added to the adjusted first global feature vector to obtain the collaborative feature expression corresponding to the visible light global feature vector; the second cross-attention output result is input into the corresponding mapping linear layer to obtain the second mapping feature vector, and the second mapping feature vector is added to the adjusted second global feature vector to obtain the collaborative feature expression corresponding to the infrared light global feature vector.

[0011] Optionally, the face liveness detection model is trained based on the following steps: constructing multiple batches of training samples based on the training sample set, and using the training samples from each of the multiple batches to train the first processing module, the second processing module, and the multi-head classification module to obtain the face liveness detection model; wherein, each batch of training samples includes: at least one real face image in visible light, at least one fake face image in visible light, at least one real face image in infrared light, and at least one fake face image in infrared light.

[0012] Optionally, the face liveness detection model further includes: an attention feature collaborative complementary fusion module for performing feature complementary fusion on the global feature vector of the visible light image extracted by the first processing module and the global feature vector of the infrared light image extracted by the second processing module; the step of training the first processing module, the second processing module, and the multi-head classification module with training samples from each of the multiple batches to obtain the face liveness detection model includes: inputting a visible light face sample image from any target batch into the first processing module to obtain feature information output by the first processing module, and inputting an infrared light face sample image from the target batch into the second processing module. In the processing module, the feature information output by the second processing module is obtained; the feature information output by the first processing module is input into the first classification head to obtain the first classification result, the second classification result, and the first distillation classification result output by the first classification head; the feature information output by the second processing module is input into the second classification head to obtain the third classification result, the fourth classification result, and the second distillation classification result output by the second classification head; and the visible light global feature vector output by the first processing module and the infrared light global feature vector output by the second processing module are input into the attention feature collaborative complementary fusion module to obtain the first collaborative feature table corresponding to the visible light global feature vector. The system calculates the first classification result and the second classification result, and inputs the first and second collaborative feature expressions into the fusion feature classification head to obtain the fifth classification result corresponding to the first collaborative feature expression and the sixth classification result corresponding to the second collaborative feature expression. Based on the first classification result output by the first classification head and the third classification result output by the second classification head, the system calculates the global classification loss corresponding to each classification result to obtain the first loss function value corresponding to each classification result. Based on the second classification result output by the first classification head and the fourth classification result output by the second classification head, the system calculates the block classification loss corresponding to each classification result to obtain the block classification loss for each classification result. The model parameters of the face liveness detection model are updated by backpropagation based on the first distillation classification result output by the first classification head and the second distillation classification result output by the second classification head, and by calculating the distillation loss corresponding to each classification result based on the first distillation classification result output by the first classification head and the second distillation classification result output by the second classification head. The model parameters of the face liveness detection model are updated by backpropagation based on the first loss function value, the second loss function value, the third loss function value and the fourth loss function value corresponding to each classification result.The first distillation classification result is obtained by inputting the first global feature vector and the first block feature vector into the first distillation classification head of the first classification head; the second distillation classification result is obtained by inputting the second global feature vector and the second block feature vector into the second distillation classification head of the second classification head.

[0013] Optionally, inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: generating a third face image based on the second face image, and inputting the second face image, the second block image corresponding to the second face image, the third face image, and the third block image corresponding to the third face image into the face liveness detection model, and generating the detection result based on the classification result of the second classification head; wherein, the third face image is a visible light image of the same size as the second face image; the third block image is an image obtained by cropping the third face image; inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: generating the detection result based on the third classification result and the fourth classification result.

[0014] This application also provides a multimodal face liveness detection device, including: An acquisition module is used to acquire a face image of a target object to be detected; the face image to be detected includes: a visible light face image and an infrared light face image; a detection module is used to input the face image to be detected into a face liveness detection model to obtain a detection result; wherein, the face liveness detection model includes: a first processing module for extracting feature information from the visible light face image, and a second processing module for extracting feature information from the infrared light face image; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face.

[0015] Optionally, the multi-head classification module includes: a first classification head for classifying the feature information extracted by the first processing module, and a second classification head for classifying the feature information extracted by the second processing module; the detection module is specifically used to input the face image to be detected, a first block image corresponding to the first face image, and a second block image corresponding to the second face image into the face liveness detection model, and generate the detection result based on the classification result of the first classification head and the classification result of the second classification head; wherein, the first face image is a visible light image; the second face image is an infrared light image; the first block image is an image obtained by cropping the first face image; the second block image is an image obtained by cropping the second face image; the classification result of the first classification head is obtained based on the feature information extracted by the first processing module; and the classification result of the second classification head is obtained based on the feature information extracted by the second processing module.

[0016] Optionally, the detection module is specifically configured to input the first face image and the first block image into the first processing module to obtain a first global feature vector for characterizing the global features of the visible light image and a first block feature vector for characterizing the block features of the visible light image; and to input the second face image and the second block image into the second processing module to obtain a second global feature vector for characterizing the global features of the infrared light image and a second block feature vector for characterizing the block features of the infrared light image.

[0017] Optionally, the face classification result includes: a first classification result, a second classification result, a third classification result, a fourth classification result, a fifth classification result, and a sixth classification result; the detection module is specifically configured to input the first global feature vector into the global feature classification head in the first classification head to obtain the first classification result; and input the first block feature vector into the block feature classification head in the first classification head to obtain the second classification result; and input the second global feature vector into the global feature classification head in the second classification head to obtain the third classification result; and input the second block feature vector into the block feature classification head in the second classification head. The detection module is further configured to obtain the fourth classification result by inputting the first global feature vector and the second global feature vector into the attention feature collaborative complementary fusion module to obtain the first collaborative feature expression corresponding to the first global feature vector and the second collaborative feature expression corresponding to the second global feature vector, and input the first collaborative feature expression and the second collaborative feature expression into the fusion feature classification head to obtain the fifth classification result and the sixth classification result; the detection module is further configured to generate the detection result based on the first classification result, the second classification result, the third classification result, the fourth classification result, the fifth classification result and the sixth classification result.

[0018] Optionally, the detection module is specifically configured to adjust the dimensions of the first global feature vector and the second global feature vector; the detection module is further configured to input the adjusted first global feature vector into the corresponding linear layer to obtain a first query feature vector, a first key feature vector, and a first value feature vector; and input the adjusted second global feature vector into the corresponding linear layer to obtain a second query feature vector, a second key feature vector, and a second value feature vector; the detection module is further configured to perform matrix multiplication of the first key feature vector and the second query feature vector to obtain a first result, and input the first result into a normalized exponential function for processing to obtain a first cross-attention matrix, multiply the first value feature vector by the first cross-attention matrix to obtain a first cross-attention output result; and input the second key feature vector into the first key feature vector. The vector is multiplied by the first query feature vector to obtain a second result, and the second result is input into a normalized exponential function for processing to obtain a second cross-attention matrix. The second value feature vector is multiplied by the second cross-attention matrix to obtain a second cross-attention output result. The detection module is further configured to input the first cross-attention output result into the corresponding mapping linear layer to obtain a first mapping feature vector, and add the first mapping feature vector to the adjusted first global feature vector to obtain the collaborative feature expression corresponding to the visible light global feature vector; and input the second cross-attention output result into the corresponding mapping linear layer to obtain a second mapping feature vector, and add the second mapping feature vector to the adjusted second global feature vector to obtain the collaborative feature expression corresponding to the infrared light global feature vector.

[0019] Optionally, the apparatus further includes: a training module; the training module is configured to construct multiple batches of training samples based on the training sample set, and use the training samples from each of the multiple batches to train the first processing module, the second processing module, and the multi-head classification module to obtain the face liveness detection model; wherein each batch of training samples includes: at least one real visible light face image, at least one fake visible light face image, at least one real infrared face image, and at least one fake infrared face image.

[0020] Optionally, the face liveness detection model further includes: an attention feature collaborative complementary fusion module for performing feature complementary fusion on the global feature vector of the visible light image extracted by the first processing module and the global feature vector of the infrared light image extracted by the second processing module; the training module is specifically used to input visible light face sample images from any target batch into the first processing module to obtain feature information output by the first processing module, and to input infrared light face sample images from the target batch into the second processing module to obtain feature information output by the second processing module; the training module is further specifically used to input the feature information output by the first processing module into the first classification head to obtain the first classification head. The first classification result, second classification result, and first distillation classification result output by the class head; and the feature information output by the second processing module is input into the second classification head to obtain the third classification result, fourth classification result, and second distillation classification result output by the second classification head; and the visible light global feature vector output by the first processing module and the infrared light global feature vector output by the second processing module are input into the attention feature collaborative complementary fusion module to obtain the first collaborative feature expression corresponding to the visible light global feature vector and the second collaborative feature expression corresponding to the infrared light global feature vector, and the first collaborative feature expression and the second collaborative feature expression are respectively input into the fusion feature classification head to obtain the... The training module is further configured to: calculate the global classification loss corresponding to each classification result based on the first classification result output by the first classification head and the third classification result output by the second classification head, to obtain the first loss function value corresponding to each classification result; calculate the block classification loss corresponding to each classification result based on the second classification result output by the first classification head and the fourth classification result output by the second classification head, to obtain the second loss function value corresponding to each classification result; and calculate the block classification loss corresponding to each classification result based on the first distillation classification result output by the first classification head and the second distillation classification result output by the second classification head, to obtain the second loss function value corresponding to each classification result; and calculate the block classification loss corresponding to each classification result based on the first distillation classification result output by the first classification head and the second distillation classification result output by the second classification head. The training module is further configured to perform backpropagation based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value for each classification result, to obtain a third loss function value for each classification result. Specifically, the training module is used to update the model parameters of the face liveness detection model by performing backpropagation based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value for each classification result. The first distillation classification result is obtained by inputting the first global feature vector and the first block feature vector into the first distillation classification head of the first classification head.The second distillation classification result is obtained by inputting the second global feature vector and the second block feature vector into the second distillation classification head of the second classification head.

[0021] Optionally, the detection module is specifically configured to generate a third face image based on the second face image, and input the second face image, the second block image corresponding to the second face image, the third face image, and the third block image corresponding to the third face image into the face liveness detection model, and generate the detection result based on the classification result of the second classification head; wherein, the third face image is a visible light image of the same size as the second face image; the third block image is an image obtained by cropping the third face image; the detection module is further configured to generate the detection result based on the third classification result and the fourth classification result.

[0022] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the above-described multimodal face liveness detection methods.

[0023] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described multimodal face liveness detection methods.

[0024] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described multimodal face liveness detection methods.

[0025] The multimodal face liveness detection method, apparatus, and electronic device provided in this application first acquire a face image of a target object to be detected; the face image to be detected includes: a visible light face image and an infrared light face image; then, the face image to be detected is input into a face liveness detection model to obtain a detection result; wherein, the face liveness detection model includes: a first processing module for extracting feature information from the visible light face image, and a second processing module for extracting feature information from the infrared light face image; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face. Thus, it can not only improve robustness to different lighting conditions and different attack types, but also greatly improve the accuracy of face liveness detection. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the multimodal face liveness detection method provided in this application; Figure 2 This is a schematic diagram of the face liveness detection model provided in this application; Figure 3 This is a schematic diagram of the structure of the Global Head and PatchHead provided in this application; Figure 4 This is a schematic diagram of the structure of the Distiller Head provided in this application; Figure 5 This is a schematic diagram of the structure of the Cross Head for fusion feature classification provided in this application; Figure 6 This is a schematic diagram of the structure of the attention feature collaborative complementary fusion module provided in this application; Figure 7 This is a schematic diagram of the structure of the multimodal face liveness detection device provided in this application; Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] The following is a detailed description of the professional fields involved in the embodiments of this application: Face liveness detection refers to methods that determine the true physiological features of a face through facial videos or images. Spoofing attacks refer to attacks on facial recognition systems such as facial access control and facial payment systems using facial forgery methods such as images, videos, and masks. Multimodal images refer to images captured by different cameras or sensors, such as infrared and visible light images. Cross-attention refers to serializing the features of different modal images and calculating the attention between two sequences.

[0031] To address the aforementioned technical problems in related technologies, this application provides a multimodal face liveness detection method. This method first uses a binocular camera to acquire real and fake face data, obtaining multimodal real and fake face images containing visible light and infrared data. The acquired face images are randomly segmented into blocks of different sizes. Then, the infrared face image and its corresponding face segmentation block, and the visible light face sample image and its corresponding face segmentation block are respectively input into a convolutional neural network for training until the network converges. This invention uses multimodal face images for liveness detection, which can improve the model's robustness to different lighting scenarios and attack types. The proposed attention feature collaborative complementary fusion module can effectively fuse visible light and infrared face features. Training the network using face images and corresponding segmentation blocks allows the network to learn local and global collaborative features. The proposed image block classification loss can constrain the consistency of features between image blocks of the same face. The proposed self-supervised distillation loss and block contrast loss enable the network to extract common information from global and local features, constraining the consistency of global and local features. Ultimately, it solves the problems of poor robustness of the model to different lighting scenarios and attack types, and low accuracy of various 3D masks.

[0032] The multimodal face liveness detection method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0033] like Figure 1As shown in the embodiment of this application, a multimodal face liveness detection method is provided, which may include the following steps 101 and 102: Step 101: Obtain the face image of the target object to be detected.

[0034] The face images to be detected include: visible light face images and infrared light face images.

[0035] For example, the target object is an object that needs to be recognized by face recognition. After the face image of the target object to be detected is collected, in addition to face recognition, face liveness detection is also required to ensure that the face collected by the camera is a real live object.

[0036] Step 102: Input the face image to be detected into the face liveness detection model to obtain the detection result.

[0037] The face liveness detection model includes: a first processing module for extracting feature information from visible light face images, and a second processing module for extracting feature information from infrared light face images; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face.

[0038] For example, after obtaining the face image of the target object to be detected, the face image to be detected can be input into the face liveness detection model to perform face liveness detection, and the detection result corresponding to the face image to be detected can be obtained. Then, based on the detection result, it can be determined whether the face image to be detected is a real face image.

[0039] Optionally, in this embodiment, the face image to be detected for the target object can include the following three cases: simultaneously containing visible light and infrared light face images, containing only visible light face images, and containing only infrared light face images. The face liveness detection model can employ different methods for detection depending on the specific case.

[0040] For example, the multi-head classification module in this application embodiment includes: a first classification head for classifying the feature information extracted by the first processing module, and a second classification head for classifying the feature information extracted by the second processing module.

[0041] Specifically, step 102 above may also include the following step 102a1: Step 102a1: Input the face image to be detected, the first block image corresponding to the first face image, and the second block image corresponding to the second face image into the face liveness detection model, and generate the detection result based on the classification result of the first classification head and the classification result of the second classification head.

[0042] Wherein, the first face image is a visible light image; the second face image is an infrared light image; the first segmented image is an image obtained by cropping the first face image; the second segmented image is an image obtained by cropping the second face image; the classification result of the first classification head is obtained based on the feature information extracted by the first processing module; the classification result of the second classification head is obtained based on the feature information extracted by the second processing module.

[0043] It should be noted that the above steps can be performed by cropping the face image into four equal parts, or by other methods.

[0044] For example, when the face liveness detection model detects both visible light face images and infrared light face images simultaneously, the face liveness detection model simultaneously determines whether the face is a live or fake based on the classification results output by the first and second classification heads.

[0045] It should be noted that, in this embodiment, the first classification head includes a global feature classification head, a block feature classification head, and a distillation feature classification head. Each classification head can output a classification result; that is, the classification result output by the first classification head includes the classification results output by the global feature classification head, the block feature classification head, and the distillation feature classification head. Similarly, the second classification head includes a global feature classification head, a block feature classification head, and a distillation feature classification head. Each classification head can output a classification result; that is, the classification result output by the second classification head includes the classification results output by the global feature classification head, the block feature classification head, and the distillation feature classification head.

[0046] For example, in order to obtain the classification result output by each classification head, feature information needs to be extracted from the visible light face image and the infrared light face image through the first processing module and the second processing module mentioned above, and the extracted feature information is input into the corresponding classification head to obtain the classification result output by each classification head.

[0047] Specifically, step 102 above may also include step 102b: Step 102b: Input the first face image and the first block image into the first processing module to obtain a first global feature vector for characterizing the global features of the visible light image and a first block feature vector for characterizing the block features of the visible light image; and input the second face image and the second block image into the second processing module to obtain a second global feature vector for characterizing the global features of the infrared light image and a second block feature vector for characterizing the block features of the infrared light image.

[0048] For example, after obtaining the visible light face image and the infrared light face image, it is necessary to crop the corresponding segmented images from the visible light face image and the infrared light face image, respectively. Then, the visible light face image and its corresponding segmented image are input into the first processing module for feature extraction to obtain a first global feature vector and a first segmented feature vector. At the same time, the infrared light face image and its corresponding segmented image are input into the second processing module for feature extraction to obtain a second global feature vector and a second segmented feature vector.

[0049] It should be noted that the first processing module and the second processing module in this embodiment are two convolutional neural network models of the same depth, and their structure can use MobileNet_V3. The above-mentioned image cropping method can be random or equally divided, and can be adjusted according to the actual situation.

[0050] For example, such as Figure 2 As shown, this embodiment of the face liveness detection model provides the following steps: A visible light face image (Global) and a visible light patch image (Patch) are input into the Encoder RGB model (i.e., the first processing module mentioned above) for feature extraction, resulting in Global_RGB (i.e., the first global feature vector) and Patch_RGB (i.e., the first patch feature vector). Simultaneously, an infrared face image (Global) and an infrared patch image (Patch) are input into the Encoder IR model (i.e., the second processing module mentioned above) for feature extraction, resulting in Global_IR (i.e., the second global feature vector) and Patch_IR (i.e., the second patch feature vector).

[0051] For example, the face classification results mentioned above include: a first classification result output by the global feature classification head in the first classification head, a second classification result output by the block feature classification head in the first classification head, a third classification result output by the distillation feature classification head in the first classification head, a fourth classification result output by the global feature classification head in the second classification head, a fifth classification result output by the block feature classification head in the second classification head, and a sixth classification result output by the distillation feature classification head in the second classification head.

[0052] For example, when using multimodal face images for liveness detection, directly stitching together the features of the multimodal images cannot effectively fuse information between different modalities, severely affecting the training results of the network. Therefore, this application provides an Attention Feature Collaborative Complementary Fusion Module (AFCFM). This module overcomes the discrimination defects of single-modal features under different spoofing attacks by weight sharing and feature complementarity fusion, achieving complementary gains. By weighting different modal features at corresponding positions on the face, face feature alignment can be achieved, enabling collaborative representation of each modality's face features and outputting their respective attention feature expressions.

[0053] For example, the face liveness detection model further includes an attention feature collaborative complementary fusion module for performing feature complementary fusion on the global feature vector of the visible light image extracted by the first processing module and the global feature vector of the infrared light image extracted by the second processing module.

[0054] Specifically, after step 102b above, step 102 may further include steps 102c1 to 102c6: Step 102c1: Input the first global feature vector into the global feature classification head in the first classification head to obtain the first classification result.

[0055] Step 102c2: Input the first block feature vector into the block feature classification head in the first classification head to obtain the second classification result.

[0056] Step 102c3: Input the second global feature vector into the global feature classification head in the second classification head to obtain the third classification result.

[0057] Step 102c4: Input the second block feature vector into the block feature classification head in the second classification head to obtain the fourth classification result.

[0058] Step 102c5: Input the first global feature vector and the second global feature vector into the attention feature collaborative complementary fusion module to obtain the first collaborative feature expression corresponding to the first global feature vector and the second collaborative feature expression corresponding to the second global feature vector. Then, input the first collaborative feature expression and the second collaborative feature expression into the fusion feature classification head to obtain the fifth classification result and the sixth classification result.

[0059] For example, after obtaining the feature information output by the first processing module and the second processing module, it can be input into the corresponding classification head to obtain the classification result output by each classification head, and the judgment of liveness and spurs can be made based on the classification result output by each classification head.

[0060] For example, such as Figure 2 As shown, the Global Head is the global feature classification head, the Patch Head is the patch feature classification head, and the Cross Head is the fused feature classification head.

[0061] The feature vector Global_RGB output by the Encoder_RGB model is input into the corresponding Global Head (i.e., the first global feature classification head mentioned above) to obtain the first classification result; the feature vector Patch_RGB output by the Encoder_RGB model is input into the corresponding Patch Head (i.e., the first block feature classification head mentioned above) to obtain the second classification result.

[0062] The feature vector Global_IR output by the Encoder_IR model is input into the corresponding Global Head (i.e., the second global feature classification head mentioned above) to obtain the third classification result; the feature vector Patch_IR output by the Encoder_IR model is input into the corresponding Patch Head (i.e., the second block feature classification head mentioned above) to obtain the fourth classification result.

[0063] The feature vectors Global_RGB (i.e., the first global feature vector mentioned above) and Global_IR (i.e., the second global feature vector mentioned above) are input into the Attention Feature Co-complementary Fusion Module (AFCFM) to obtain the co-feature representations corresponding to each global feature vector. Then, the two co-feature representations are input into the fusion feature head to obtain the classification results corresponding to each global feature vector, namely the fifth and sixth classification results mentioned above.

[0064] Specifically, the step of generating collaborative feature representation through the attention feature collaborative complementary fusion module in step 102c5 above may include the following steps 102c51 to 102c54: Step 102c51: Adjust the dimensions of the first global feature vector and the second global feature vector.

[0065] Step 102c52: Input the adjusted first global feature vector into the corresponding linear layer to obtain the first query feature vector, the first key feature vector, and the first value feature vector; and input the adjusted second global feature vector into the corresponding linear layer to obtain the second query feature vector, the second key feature vector, and the second value feature vector.

[0066] Step 102c53: Multiply the first key feature vector and the second query feature vector by a matrix to obtain a first result, and input the first result into a normalized exponential function for processing to obtain a first cross-attention matrix. Multiply the first value feature vector by the first cross-attention matrix to obtain a first cross-attention output result. Then, multiply the second key feature vector and the first query feature vector by a matrix to obtain a second result, and input the second result into a normalized exponential function for processing to obtain a second cross-attention matrix. Multiply the second value feature vector by the second cross-attention matrix to obtain a second cross-attention output result.

[0067] Step 102c54: Input the first cross-attention output result into the corresponding mapping linear layer to obtain the first mapping feature vector, and add the first mapping feature vector to the adjusted first global feature vector to obtain the collaborative feature expression corresponding to the visible light global feature vector; and input the second cross-attention output result into the corresponding mapping linear layer to obtain the second mapping feature vector, and add the second mapping feature vector to the adjusted second global feature vector to obtain the collaborative feature expression corresponding to the infrared light global feature vector.

[0068] It should be noted that the steps for generating the collaborative feature representation corresponding to each global feature vector by the aforementioned attention feature collaborative complementary fusion module will be explained in detail during the model training process.

[0069] For example, after obtaining the classification results output by each classification head, the determination of whether a body is alive or a prosthesis can be made based on the obtained classification results.

[0070] Specifically, after steps 102c1 to 102c5, step 102 may further include the following step 102d1: Step 102d1: Generate the detection result based on the first classification result, the second classification result, the third classification result, the fourth classification result, the fifth classification result, and the sixth classification result.

[0071] For example, when the face liveness detection model detects both visible light and infrared light face images simultaneously, the input to the model is a dual-path face image consisting of both visible light and infrared light images. By inputting these dual-path face images into the model, the global feature classification probabilities (Prob) of the visible light and infrared light face images are obtained. rgb-global and Prob ir-global (i.e., the first and third classification results mentioned above), the block feature classification probability Prob rgb-patch and Prob ir-patch(i.e., the second and fourth classification results mentioned above), and the fusion feature classification probability Prob rgb-cross and Prob ir-cross (That is, the fifth and sixth classification results mentioned above). Then, the probabilities extracted above can be used to determine whether the body is a live or a prosthesis based on the following formula: (Formula 1)

[0072] in, .

[0073] In one possible implementation, the image used for face liveness detection may only contain an infrared face image. In this case, a visible light face image can be generated first based on the infrared face image, and then the two images can be input into the face liveness detection model for detection.

[0074] Specifically, step 201 above may also include the following step 102a2: Step 102a2: If the face image to be detected contains only the second face image, generate a third face image based on the second face image, and input the second face image, the second block image corresponding to the second face image, the third face image, and the third block image corresponding to the third face image into the face liveness detection model, and generate the detection result based on the classification result of the second classification head.

[0075] For example, when the face liveness detection model only detects an infrared face image, it needs to generate a visible light face image of the same size with a value of 0 based on the infrared face image, and then input both images into the face liveness detection model. However, the face liveness detection model only determines whether an image is a live or fake based on the classification results output by the second classification head mentioned above.

[0076] It should be noted that, in order to maintain the integrity of the model's forward inference, the model can be used to determine the authenticity of a liveness detection model in scenarios without visible light, such as at night, in dark rooms, or in corridors, by using infrared images and constructed visible light images. This expands the usage environment of the device. When only infrared face images are detected, a visible light face image needs to be generated based on the infrared face image, and both the infrared face image and the generated visible light face image should be input into the face liveness detection model.

[0077] For example, when the face liveness detection model only detects a face image in visible light, it skips the face liveness detection process directly.

[0078] Specifically, based on step 102a2 above, step 102 may further include the following step 102d2: Step 102d2: Generate the detection result based on the third classification result and the fourth classification result.

[0079] For example, when the face liveness detection model only detects an infrared face image, a visible light face image of the same size but with a value of 0 is generated. The generated visible light face image, the detected infrared face image, and their corresponding segmented images are then input into the face liveness detection model. Finally, liveness detection is performed solely based on the classification probability of the single-channel infrared image. Specifically, the extracted probabilities can be used to determine liveness and spoofing using the following formula: (Formula 2)

[0080] Optionally, in the embodiments of this application, during the training process of the model, the corresponding loss function value can be calculated through the classification results output by each of the above classification heads, and the model parameters can be adjusted according to the loss function value.

[0081] For example, prior to step 102 above, the multimodal face liveness detection method provided in this application embodiment may further include the following step 103: Step 103: Construct multiple batches of training samples based on the training sample set, and use the training samples from each of the multiple batches to train the first processing module, the second processing module, and the multi-head classification module to obtain the face liveness detection model.

[0082] Each batch of training samples includes: at least one real face image in visible light, at least one fake face image in visible light, at least one real face image in infrared light, and at least one fake face image in infrared light.

[0083] For example, in this embodiment of the application, visible light cameras and infrared cameras are used to collect real face images and face images of various types of spoofing attacks as experimental datasets (i.e., the training sample set mentioned above) to train and validate the model. The images of spoofing attacks (i.e., the forged face sample images mentioned above) include: printed photos, cropped photos, mobile phone photos, 3D masks, headgear, and other face images of various spoofing attack types. In the training sample set mentioned above, real face images are labeled 1, and forged face images are labeled 0.

[0084] For example, after obtaining the above training sample set, the sample images need to be processed. First, the visible light face sample images (including real face images and fake face images) and infrared light face sample images (including real face images and fake face images) in the training sample set are scaled to the same size (height H). g =224, width W g=224), marked as I global After that, from I global Randomly crop N (N=2) images of size (height H) from the image. p =96, width W p Block image I (=96) patch Different image enhancement methods were used to enhance the original image I. global and block image I patch Enhancement processing is performed to obtain enhanced data.

[0085] For example, such as Figure 2 As shown, when training the face liveness detection model, the enhanced data is input into two convolutional neural network models, Encoder_RGB and Encoder_IR, to obtain the feature vectors corresponding to each enhanced data, including: Global_RGB (i.e., the first global feature vector mentioned above) and Patch_RGB (i.e., the first block feature vector mentioned above), as well as Global_IR (i.e., the second global feature vector mentioned above) and Patch_IR (i.e., the second block feature vector mentioned above).

[0086] For example, during training, the extracted feature information is input into a multi-head classification module containing a first multi-head and a second multi-head to obtain the classification output corresponding to each feature vector, thereby calculating the training loss corresponding to each output.

[0087] Specifically, step 103 above may also include the following steps: 103a, 103b, 103c, and 103d: Step 103a: Input the visible light face sample image from any target batch into the first processing module to obtain the feature information output by the first processing module, and input the infrared face sample image from the target batch into the second processing module to obtain the feature information output by the second processing module.

[0088] Step 103b: Input the feature information output by the first processing module into the first classification head to obtain the first classification result, the second classification result, and the first distillation classification result output by the first classification head; and input the feature information output by the second processing module into the second classification head to obtain the third classification result, the fourth classification result, and the second distillation classification result output by the second classification head; and input the visible light global feature vector output by the first processing module and the infrared light global feature vector output by the second processing module into the attention feature collaborative complementary fusion module to obtain the first collaborative feature expression corresponding to the visible light global feature vector and the second collaborative feature expression corresponding to the infrared light global feature vector, and input the first collaborative feature expression and the second collaborative feature expression into the fusion feature classification head respectively to obtain the fifth classification result corresponding to the first collaborative feature expression and the sixth classification result corresponding to the second collaborative feature expression.

[0089] The first distillation classification result is obtained by inputting the first global feature vector and the first block feature vector into the first distillation classification head of the first classification head; the second distillation classification result is obtained by inputting the second global feature vector and the second block feature vector into the second distillation classification head of the second classification head.

[0090] For example, such as Figure 2 As shown, the Distiller Head is the distillation classification head. The Patch_RGB and Global_RGB values ​​are input into the corresponding Distiller Head (i.e., the first distillation feature classification head) to obtain the first distillation classification result. Similarly, the Patch_IR and Global_IR values ​​are input into the corresponding Distiller Head (i.e., the second distillation feature classification head) to obtain the second distillation classification result.

[0091] Step 103c: Based on the first classification result output by the first classification head and the third classification result output by the second classification head, calculate the global classification loss corresponding to each classification result to obtain the first loss function value corresponding to each classification result; and based on the second classification result output by the first classification head and the fourth classification result output by the second classification head, calculate the block classification loss corresponding to each classification result to obtain the second loss function value corresponding to each classification result; and based on the first distillation classification result output by the first classification head and the second distillation classification result output by the second classification head, calculate the distillation loss corresponding to each classification result to obtain the third loss function value corresponding to each classification result; and based on the fifth classification result and the sixth classification result, calculate the fusion feature classification loss corresponding to each classification result to obtain the fourth loss function value corresponding to each classification result.

[0092] Step 103d: Backpropagation is performed based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value corresponding to each classification result to update the model parameters of the face liveness detection model.

[0093] For example, such as Figure 2 As shown, during training, the global feature vectors Global_RGB and Global_IR are input into the corresponding global feature classification head, Global Head, to calculate the global classification loss corresponding to the classification result output by each classification head, thus obtaining the first loss function value corresponding to each classification result. The patch feature vectors Patch_RGB and Patch_IR are input into the corresponding patch feature classification head, and the patch classification loss is calculated, thus obtaining the second loss function value corresponding to each classification result. The feature pairs composed of visible light feature vectors Global_RGB and Patch_RGB and infrared light feature vectors Global_IR and Patch_IR are input into the corresponding distillation classification heads, respectively, to calculate the corresponding self-supervised distillation loss, thus obtaining the third loss function value corresponding to each classification result. At the same time, the global feature vectors Global_RGB and Global_IR also need to be input into the attention feature collaborative complementary fusion module to calculate the classification loss of the fused features, thus obtaining the fourth loss function value corresponding to each classification result.

[0094] It should be noted that the first, second, third, and fourth loss function values ​​mentioned above are all loss function values ​​of the same type, that is, the loss function values ​​corresponding to each classification result.

[0095] For example, for the first loss function value mentioned above, the global feature vectors Global_RGB and Global_IR are sequentially input as follows: Figure 3 The Global Head shown yields classification output results (i.e., the first and third classification results mentioned above). The global classification loss for each is calculated, resulting in the first loss function value corresponding to the first classification result and the first loss function value corresponding to the second classification result (e.g., ...). Figure 2 The SSL Loss in the model (the global classification loss) can be calculated using the following formula: (Formula 3)

[0096] When the global feature vector is used as input, the dimensions of the input features are (B, C, H). gf =7,W gf =7), the dimension of the classification output class_out is (B, Cls=2), label={0,1} is the class label, and CrossEntropy is the cross-entropy loss function.

[0097] For example, in calculating the second loss function value, the block feature vectors Patch_RGB and Patch_IR are sequentially input as follows: Figure 3 The classification output results (i.e., the second and fourth classification results mentioned above) are obtained from the Patch Head shown. The respective patch classification losses (i.e., as shown in the diagram) are then calculated. Figure 2 The PatchLoss value can be calculated using the following formula: (Formula 4)

[0098] When the block feature vector is used as input, the dimension of the input feature is (2B, C, H). pf =3, W pf =3), after average pooling (Avgpool2d) and max pooling (Maxpool2d) operations, the classification result class_out is output. The dimension of class_out is (2B, Cls=2), label={0,1} is the class label, and CrossEntropy is the cross-entropy loss function. The dimension of embedding is (2B, C, 1, 1), and the embedding is divided into two parts on the first dimension to obtain embedding1 and embedding2. MSE is the mean squared error loss function, which measures the feature similarity between two image patches, and α is the weighting coefficient. Since the patch image I patchThe number of embeddings is N=2, so the dimension of the embedding is twice that of the global image, and it can be divided into two parts to calculate the loss between different image patches of the same image.

[0099] It should be noted that, in the embodiments of this application, the Global Head and Patch Head corresponding to the first processing module have the same structure and share parameters; the Global Head and Patch Head corresponding to the second processing module have the same structure and share parameters.

[0100] For example, in calculating the third loss function value mentioned above, the global feature vector and the block feature vector are paired up to obtain image feature pairs (Global_RGB, Patch_RGB) and (Global_IR, Patch_IR), which are then input into... Figure 4 In the distillation sorting head shown, the self-supervised distillation loss of each is calculated (i.e., as shown in the figure). Figure 2 The BCM Loss in BCM can be calculated using the following formula: (Formula 5)

[0101] Among them, the dimension of the global feature classification output, distil_global feature, is (B, C1, C2, ..., C3). distiller =10240), the dimension of the local classification output distiller_patch feature is (2B, Cls) distiller =10240), distiller_patch half The loss function is half the value of the local features, where Softmax represents the normalization exponential function and log is the logarithmic function. As shown in Formula 5 above, this loss function calculates the loss by distilling local features using the same global image features.

[0102] For example, regarding the calculation of the fourth loss function value, after using different image enhancement methods to enhance visible light face sample images and infrared face sample images and their corresponding block images, resulting in data with two different fields of view, distortion, and deformation types, the image-enhanced visible light and infrared data are input into two deep learning networks, Encoder_RGB and Encoder_IR, respectively, to obtain the corresponding feature vectors: Global_RGB, Global_IR, Patch_RGB, and Patch_IR. Then, the global feature vectors (Global_RGB, Global_IR) are input into the attention feature collaborative complementary fusion module to obtain the collaborative feature representations of the visible light face sample images and infrared face sample images: Residual_out1 and Residual_out2, respectively, and these are then sequentially input into... Figure 5 In the Cross Head of the fused feature classification, the fused feature classification loss is calculated to obtain the fourth loss function value mentioned above. The calculation formula is shown in Formula 6 below: (Formula 6)

[0103] The fused face feature Residual_out has dimensions (B, N=49, C), where B is the number of images in a batch, N is the product of the height and width of the sample image, and C is the number of channels. The cross_out with dimensions (B, Cls=2) is the Cross Head classification output (i.e., the seventh classification result mentioned above), where label={0, 1} is the class label, and CrossEntropy is the cross-entropy loss function used to calculate the classification loss of the feature.

[0104] like Figure 5 The diagram shows the structure of the Cross Head feature head described above. `mean(dim=1)` represents the average value calculated along the first dimension to standardize or normalize the input data. Then, a linear layer is used to generate the classification result, determining whether it is a real or fake data.

[0105] For example, after obtaining the above four loss function values, backpropagation can be performed based on each loss value to iteratively update the model parameters and train the network until the network converges.

[0106] For example, such as Figure 2 As shown, during the model testing process, the visible light and infrared test images were scaled to the same size (H). g =224, W g =224), then perform center cropping on the image, cropping to a size of (H)c =192, W c Image I (=192) crop , will I crop Divide into 4 equal sizes (H) p =96, W p Image block I (=96) patch Then, the visible light and infrared images and their corresponding image patches are input into the two deep learning networks, Encoder_RGB and Encoder_IR, respectively, to obtain the corresponding features. Finally, the features are input into the multi-head module to obtain the classification probability of each feature, and the weighted average of each classification probability is used to determine whether the liveness is real or fake.

[0107] It should be noted that in this embodiment, information from multiple dimensions is used in combination. The classification loss and self-supervised distillation loss are calculated using global features, local block features, and fused features of the face image, respectively. The block classification loss constrains the consistency of features between two image blocks of the same face image, forcing the network to learn image texture and noise, and compelling the network to pay more attention to the local facial features. The self-supervised distillation loss maps the global and local features of the face to the same high-dimensional feature space, ensuring consistency between global and local features and improving the robustness of the model.

[0108] Optionally, in this embodiment of the application, in order to overcome the discrimination defects of single modality features under different spoofing attacks and achieve complementary gains, an attention feature collaborative complementary fusion module needs to be introduced during the training process of the model.

[0109] Specifically, step 103b above may also include steps 103b1 to 103b4: Step 103b1: Adjust the dimensions of the visible light global feature vector and the infrared light global feature vector input to the attention feature collaborative complementary fusion module based on the number of images contained in the target batch and the size of each image.

[0110] Step 103b2: Input the adjusted visible light global feature vector into the corresponding linear layer to obtain the first query feature vector, the first key feature vector, and the first value feature vector; and input the adjusted infrared light global feature vector into the corresponding linear layer to obtain the second query feature vector, the second key feature vector, and the second value feature vector.

[0111] Step 103b3: Multiply the first key feature vector and the second query feature vector by a matrix to obtain a first result, and input the first result into a normalized exponential function for processing to obtain a first cross-attention matrix. Multiply the first value feature vector by the first cross-attention matrix to obtain a first cross-attention output result. Then, multiply the second key feature vector and the first query feature vector by a matrix to obtain a second result, and input the second result into a normalized exponential function for processing to obtain a second cross-attention matrix. Multiply the second value feature vector by the second cross-attention matrix to obtain a second cross-attention output result.

[0112] Step 103b4: Input the first cross-attention output result into the corresponding mapping linear layer to obtain the first mapping feature vector, and add the first mapping feature vector to the adjusted visible light global feature vector to obtain the collaborative feature expression corresponding to the visible light global feature vector; and input the second cross-attention output result into the corresponding mapping linear layer to obtain the second mapping feature vector, and add the second mapping feature vector to the adjusted infrared light global feature vector to obtain the collaborative feature expression corresponding to the infrared light global feature vector.

[0113] For example, such as Figure 6 The diagram shown illustrates the structure of the attention feature collaborative complementary fusion module. First, the dimensions of the global feature vectors Global_RGB and Global_IR are changed from (B, C, H) gf =7,W gf=7) Adjust (B, N=49, C), input the visible light global feature vector and the infrared light global feature vector into the corresponding linear layers (Linear Q1, Linear K1, Linear V1) and (Linear Q2, Linear K2, Linear V2), respectively, to obtain the corresponding query features Query1 and Query2 (i.e., the first query feature vector and the second query feature vector mentioned above), key features Key1 and Key2 (i.e., the first key feature vector and the second key feature vector mentioned above), and value features Value1 and Value2 (i.e., the first value feature vector and the second value feature vector mentioned above). Among them, the weights of the corresponding linear layers (Linear Q1, Linear K1, Linear V1) and (Linear Q2, Linear K2, Linear V2) are shared, and the dimension of the feature vector output by each linear layer is also (B, N=49, C). Next, the key feature Key1 and the query feature Query2 are multiplied by matrix, and after being processed by a normalized exponential function (e.g., the Softmax function), the cross attention matrix CAM1 is obtained. Multiply the value features Vaule1 and CAM1 by a matrix to obtain the cross-attention output Out1. Similarly, according to... Figure 6 The same calculations are performed on the right side to obtain Out2. The specific calculation steps are shown in Formula 7 below: (Formula 7)

[0114] Wherein, CAM1 and CAM2 are matrices of dimension (B, N, N), representing the weighted sum of visible light and infrared features in this patent, and calculating cross-attention in the feature space. The specific calculation steps are shown in Formula 8 below: (Formula 8)

[0115] Where Softmax represents the normalized exponential function, is matrix multiplication, and T represents the transpose operation.

[0116] Finally, Out1 is input into the mapping linear layer Linear1 to obtain the mapped feature Project1. The original feature F1 and Project1 are added together to obtain the residual output Residual_out1 of the cross-attention (i.e., the output result of the first cross-attention mentioned above). Similarly, Figure 6 The steps on the right undergo the same calculations to obtain Residual_out2 (i.e., the output of the second cross-attention mentioned above). The specific calculation steps are shown in Formula Nine below: (Formula Nine)

[0117] It should be noted that, unlike existing technologies that obtain fused features through feature splicing or weighting, the attention feature collaborative complementary fusion module in this embodiment achieves facial feature alignment by weighting different modal features at corresponding positions on the face. This enables collaborative representation of facial features across different modalities and outputs their respective attention feature expressions. Through weight sharing and complementary fusion of modal features, the module overcomes the discrimination defects of single-modal features under different spoofing attacks, achieving complementary gains and improving the accuracy of liveness detection under different lighting conditions.

[0118] This application presents two face image segmentation strategies and their corresponding usage methods. During training, the face image is randomly segmented into blocks to calculate the local consistency loss between image blocks and the consistency loss between image blocks and the global image. During testing, the face image is centrally cropped and then evenly segmented, ensuring that each face image block contains more facial information and less boundary information. The classification probability of each image block, the global classification probability, and the classification probability of the fused features are calculated to obtain a comprehensive classification result, which is more reliable than the classification result obtained from a single image.

[0119] In this embodiment, losses are calculated for the global, local, and fused features of the face image, enabling the network to learn both global and local information of the face image simultaneously. The image patch classification loss constrains the feature consistency between two image patches of the same face image, forcing the network to learn the texture and noise of the image and to pay more attention to the local facial features. The self-supervised distillation loss maps the global and local features of the face to the same high-dimensional feature space, ensuring consistency between global and local features and improving the robustness of the model.

[0120] The multimodal face liveness detection method provided in this application first acquires a face image of the target object to be detected; the face image to be detected includes: a visible light face image and an infrared light face image; then, the face image to be detected is input into a face liveness detection model to obtain a detection result; wherein, the face liveness detection model includes: a first processing module for extracting feature information from the visible light face image, and a second processing module for extracting feature information from the infrared light face image; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face. Thus, it can not only improve robustness to different lighting conditions and different attack types, but also greatly improve the accuracy of face liveness detection.

[0121] It should be noted that the multimodal face liveness detection method provided in this application embodiment can be executed by a multimodal face liveness detection device, or a control module within that device for executing the multimodal face liveness detection method. This application embodiment uses the execution of the multimodal face liveness detection method by a multimodal face liveness detection device as an example to illustrate the multimodal face liveness detection device provided in this application embodiment.

[0122] It should be noted that, in the embodiments of this application, the multimodal face liveness detection methods shown in the accompanying drawings are all illustrated using one accompanying drawing from one of the embodiments of this application as an example. In specific implementation, the multimodal face liveness detection methods shown in the accompanying drawings of the above methods can also be implemented in conjunction with any other accompanying drawings shown in the above embodiments, which will not be elaborated here.

[0123] The multimodal face liveness detection device provided in this application is described below. The multimodal face liveness detection method described below can be referred to in correspondence with the multimodal face liveness detection method described above.

[0124] Figure 7 This is a schematic diagram of the structure of the multimodal face liveness detection device provided in the embodiments of this application, as shown below. Figure 7 As shown, it specifically includes: An acquisition module 701 is used to acquire a face image of a target object to be detected; the face image to be detected includes a visible light face image and an infrared light face image; a detection module 702 is used to input the face image to be detected into a face liveness detection model to obtain a detection result; wherein, the face liveness detection model includes: a first processing module for extracting feature information from the visible light face image, and a second processing module for extracting feature information from the infrared light face image; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face.

[0125] Optionally, the multi-head classification module includes: a first classification head for classifying the feature information extracted by the first processing module, and a second classification head for classifying the feature information extracted by the second processing module; the detection module 702 is specifically used to input the face image to be detected, the first block image corresponding to the first face image, and the second block image corresponding to the second face image into the face liveness detection model, and generate the detection result based on the classification result of the first classification head and the classification result of the second classification head; wherein, the first face image is a visible light image; the second face image is an infrared light image; the first block image is an image obtained by cropping the first face image; the second block image is an image obtained by cropping the second face image; the classification result of the first classification head is obtained based on the feature information extracted by the first processing module; the classification result of the second classification head is obtained based on the feature information extracted by the second processing module.

[0126] Optionally, the detection module 702 is specifically configured to input the first face image and the first block image into the first processing module to obtain a first global feature vector for characterizing the global features of the visible light image and a first block feature vector for characterizing the block features of the visible light image; and to input the second face image and the second block image into the second processing module to obtain a second global feature vector for characterizing the global features of the infrared light image and a second block feature vector for characterizing the block features of the infrared light image.

[0127] Optionally, the face classification result includes: a first classification result, a second classification result, a third classification result, a fourth classification result, a fifth classification result, and a sixth classification result; the detection module 702 is specifically used to input the first global feature vector into the global feature classification head in the first classification head to obtain the first classification result; and to input the first block feature vector into the block feature classification head in the first classification head to obtain the second classification result; and to input the second global feature vector into the global feature classification head in the second classification head to obtain the third classification result; and to input the second block feature vector into the block feature classification head in the second classification head. In the process, the fourth classification result is obtained; and the first global feature vector and the second global feature vector are input into the attention feature collaborative complementary fusion module to obtain the first collaborative feature expression corresponding to the first global feature vector and the second collaborative feature expression corresponding to the second global feature vector, and the first collaborative feature expression and the second collaborative feature expression are respectively input into the fusion feature classification head to obtain the fifth classification result and the sixth classification result; the detection module 702 is further specifically used to generate the detection result based on the first classification result, the second classification result, the third classification result, the fourth classification result, the fifth classification result and the sixth classification result.

[0128] Optionally, the detection module 702 is specifically used to adjust the dimensions of the first global feature vector and the second global feature vector; the detection module 702 is further used to input the adjusted first global feature vector into the corresponding linear layer to obtain a first query feature vector, a first key feature vector, and a first value feature vector; and to input the adjusted second global feature vector into the corresponding linear layer to obtain a second query feature vector, a second key feature vector, and a second value feature vector; the detection module 702 is further used to perform matrix multiplication of the first key feature vector and the second query feature vector to obtain a first result, and to input the first result into a normalized exponential function for processing to obtain a first cross-attention matrix, and to multiply the first value feature vector with the first cross-attention matrix to obtain a first cross-attention output result; and to input the second The key feature vector is matrix-multiplied with the first query feature vector to obtain a second result, and the second result is input into a normalized exponential function for processing to obtain a second cross-attention matrix. The second value feature vector is multiplied with the second cross-attention matrix to obtain a second cross-attention output result. The detection module 702 is further configured to input the first cross-attention output result into the corresponding mapping linear layer to obtain a first mapping feature vector, and add the first mapping feature vector to the adjusted first global feature vector to obtain the collaborative feature expression corresponding to the visible light global feature vector; and input the second cross-attention output result into the corresponding mapping linear layer to obtain a second mapping feature vector, and add the second mapping feature vector to the adjusted second global feature vector to obtain the collaborative feature expression corresponding to the infrared light global feature vector.

[0129] Optionally, the apparatus further includes: a training module; the training module is configured to construct multiple batches of training samples based on the training sample set, and use the training samples from each of the multiple batches to train the first processing module, the second processing module, and the multi-head classification module to obtain the face liveness detection model; wherein each batch of training samples includes: at least one real visible light face image, at least one fake visible light face image, at least one real infrared face image, and at least one fake infrared face image.

[0130] Optionally, the face liveness detection model further includes: an attention feature collaborative complementary fusion module for performing feature complementary fusion on the global feature vector of the visible light image extracted by the first processing module and the global feature vector of the infrared light image extracted by the second processing module; the training module is specifically used to input visible light face sample images from any target batch into the first processing module to obtain feature information output by the first processing module, and to input infrared light face sample images from the target batch into the second processing module to obtain feature information output by the second processing module; the training module is further specifically used to input the feature information output by the first processing module into the first classification head to obtain the first classification head. The first classification result, second classification result, and first distillation classification result output by the class head; and the feature information output by the second processing module is input into the second classification head to obtain the third classification result, fourth classification result, and second distillation classification result output by the second classification head; and the visible light global feature vector output by the first processing module and the infrared light global feature vector output by the second processing module are input into the attention feature collaborative complementary fusion module to obtain the first collaborative feature expression corresponding to the visible light global feature vector and the second collaborative feature expression corresponding to the infrared light global feature vector, and the first collaborative feature expression and the second collaborative feature expression are respectively input into the fusion feature classification head to obtain the... The training module is further configured to: calculate the global classification loss corresponding to each classification result based on the first classification result output by the first classification head and the third classification result output by the second classification head, to obtain the first loss function value corresponding to each classification result; calculate the block classification loss corresponding to each classification result based on the second classification result output by the first classification head and the fourth classification result output by the second classification head, to obtain the second loss function value corresponding to each classification result; and calculate the block classification loss corresponding to each classification result based on the first distillation classification result output by the first classification head and the second distillation classification result output by the second classification head, to obtain the second loss function value corresponding to each classification result; and calculate the block classification loss corresponding to each classification result based on the first distillation classification result output by the first classification head and the second distillation classification result output by the second classification head. The training module is further configured to perform backpropagation based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value for each classification result, to obtain a third loss function value for each classification result. Specifically, the training module is used to update the model parameters of the face liveness detection model by performing backpropagation based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value for each classification result. The first distillation classification result is obtained by inputting the first global feature vector and the first block feature vector into the first distillation classification head of the first classification head.The second distillation classification result is obtained by inputting the second global feature vector and the second block feature vector into the second distillation classification head of the second classification head.

[0131] Optionally, the detection module 702 is specifically used to generate a third face image based on the second face image, and input the second face image, the second block image corresponding to the second face image, the third face image, and the third block image corresponding to the third face image into the face liveness detection model, and generate the detection result based on the classification result of the second classification head; wherein, the third face image is a visible light image of the same size as the second face image; the third block image is an image obtained by cropping the third face image; the detection module 702 is further used to generate the detection result based on the third classification result and the fourth classification result.

[0132] The multimodal face liveness detection device provided in this application first acquires a face image of a target object to be detected; the face image to be detected includes a visible light face image and an infrared light face image; then, the face image to be detected is input into a face liveness detection model to obtain a detection result; wherein, the face liveness detection model includes: a first processing module for extracting feature information from the visible light face image, and a second processing module for extracting feature information from the infrared light face image; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face. Thus, it can not only improve robustness to different lighting conditions and different attack types, but also greatly improve the accuracy of face liveness detection.

[0133] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can call logic instructions in the memory 830 to execute a multimodal face liveness detection method. This method includes: acquiring a face image of a target object to be detected; the face image to be detected includes a visible light face image and an infrared light face image; inputting the face image to be detected into a face liveness detection model to obtain a detection result; wherein the face liveness detection model includes: a first processing module for extracting feature information from the visible light face image, and a second processing module for extracting feature information from the infrared light face image; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face.

[0134] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] On the other hand, this application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the multimodal face liveness detection method provided by the above methods. The method includes: acquiring a face image of a target object to be detected; the face image to be detected includes: a visible light face image and an infrared light face image; inputting the face image to be detected into a face liveness detection model to obtain a detection result; wherein, the face liveness detection model includes: a first processing module for extracting feature information from the visible light face image, and a second processing module for extracting feature information from the infrared light face image; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face.

[0136] Furthermore, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the aforementioned multimodal face liveness detection methods. The method includes: acquiring a face image of a target object to be detected; the face image to be detected includes: a visible light face image and an infrared light face image; inputting the face image to be detected into a face liveness detection model to obtain a detection result; wherein the face liveness detection model includes: a first processing module for extracting feature information from the visible light face image, and a second processing module for extracting feature information from the infrared light face image; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal face liveness detection method, characterized in that, include: Obtain the image of the face to be detected from the target object; The face images to be detected include: visible light face images and infrared light face images; The face image to be detected is input into the face liveness detection model to obtain the detection result; The face liveness detection model includes: a first processing module for extracting feature information from visible light face images, and a second processing module for extracting feature information from infrared light face images; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face.

2. The method according to claim 1, characterized in that, The multi-head classification module includes: a first classification head for classifying the feature information extracted by the first processing module, and a second classification head for classifying the feature information extracted by the second processing module; The step of inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: The face image to be detected, the first block image corresponding to the first face image, and the second block image corresponding to the second face image are input into the face liveness detection model, and the detection result is generated based on the classification result of the first classification head and the classification result of the second classification head. Wherein, the first face image is a visible light image; the second face image is an infrared light image; the first segmented image is an image obtained by cropping the first face image; the second segmented image is an image obtained by cropping the second face image; the classification result of the first classification head is obtained based on the feature information extracted by the first processing module; the classification result of the second classification head is obtained based on the feature information extracted by the second processing module.

3. The method according to claim 2, characterized in that, The step of inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: The first face image and the first block image are input into the first processing module to obtain a first global feature vector for characterizing the global features of the visible light image and a first block feature vector for characterizing the block features of the visible light image; and the second face image and the second block image are input into the second processing module to obtain a second global feature vector for characterizing the global features of the infrared light image and a second block feature vector for characterizing the block features of the infrared light image.

4. The method according to claim 3, characterized in that, The face classification results include: a first classification result, a second classification result, a third classification result, a fourth classification result, a fifth classification result, and a sixth classification result; The step of inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: The first global feature vector is input into the global feature classification head in the first classification head to obtain the first classification result; and the first block feature vector is input into the block feature classification head in the first classification head to obtain the second classification result; and the second global feature vector is input into the global feature classification head in the second classification head to obtain the third classification result; and the second block feature vector is input into the block feature classification head in the second classification head to obtain the fourth classification result; and the first global feature vector and the second global feature vector are input into the attention feature collaborative complementary fusion module to obtain the first collaborative feature expression corresponding to the first global feature vector and the second collaborative feature expression corresponding to the second global feature vector, and the first collaborative feature expression and the second collaborative feature expression are respectively input into the fusion feature classification head to obtain the fifth classification result and the sixth classification result; The step of inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: The detection result is generated based on the first classification result, the second classification result, the third classification result, the fourth classification result, the fifth classification result, and the sixth classification result.

5. The method according to claim 4, characterized in that, The step of inputting the first global feature vector and the second global feature vector into the attention feature collaborative complementary fusion module to obtain the first collaborative feature expression corresponding to the first global feature vector and the second collaborative feature expression corresponding to the second global feature vector includes: Adjust the dimensions of the first global feature vector and the second global feature vector; The adjusted first global feature vector is input into the corresponding linear layer to obtain the first query feature vector, the first key feature vector, and the first value feature vector; and the adjusted second global feature vector is input into the corresponding linear layer to obtain the second query feature vector, the second key feature vector, and the second value feature vector. The first key feature vector and the second query feature vector are multiplied by a matrix to obtain a first result. The first result is then processed by a normalized exponential function to obtain a first cross-attention matrix. The first value feature vector is multiplied by the first cross-attention matrix to obtain a first cross-attention output result. The second key feature vector and the first query feature vector are multiplied by a matrix to obtain a second result. The second result is then processed by a normalized exponential function to obtain a second cross-attention matrix. The second value feature vector is then multiplied by the second cross-attention matrix to obtain a second cross-attention output result. The first cross-attention output is input into the corresponding mapping linear layer to obtain the first mapping feature vector. The first mapping feature vector is then added to the adjusted first global feature vector to obtain the collaborative feature expression corresponding to the visible light global feature vector. The second cross-attention output is input into the corresponding mapping linear layer to obtain the second mapping feature vector. The second mapping feature vector is then added to the adjusted second global feature vector to obtain the collaborative feature expression corresponding to the infrared light global feature vector.

6. The method according to any one of claims 2 to 5, characterized in that, The face liveness detection model was trained based on the following steps: Based on the training sample set, multiple batches of training samples are constructed, and the first processing module, the second processing module, and the multi-head classification module are trained using the training samples from each of the multiple batches to obtain the face liveness detection model. Each batch of training samples includes: at least one real face image in visible light, at least one fake face image in visible light, at least one real face image in infrared light, and at least one fake face image in infrared light.

7. The method according to claim 6, characterized in that, The face liveness detection model further includes: an attention feature collaborative complementary fusion module for performing feature complementary fusion on the global feature vector of the visible light image extracted by the first processing module and the global feature vector of the infrared light image extracted by the second processing module; The step of training the first processing module, the second processing module, and the multi-head classification module using training samples from each of the multiple batches to obtain the face liveness detection model includes: The visible light face sample image from any target batch is input into the first processing module to obtain the feature information output by the first processing module, and the infrared light face sample image from the target batch is input into the second processing module to obtain the feature information output by the second processing module. The feature information output by the first processing module is input into the first classification head to obtain the first classification result, the second classification result, and the first distillation classification result output by the first classification head; and the feature information output by the second processing module is input into the second classification head to obtain the third classification result, the fourth classification result, and the second distillation classification result output by the second classification head; and the visible light global feature vector output by the first processing module and the infrared light global feature vector output by the second processing module are input into the attention feature collaborative complementary fusion module to obtain the first collaborative feature expression corresponding to the visible light global feature vector and the second collaborative feature expression corresponding to the infrared light global feature vector, and the first collaborative feature expression and the second collaborative feature expression are respectively input into the fusion feature classification head to obtain the fifth classification result corresponding to the first collaborative feature expression and the sixth classification result corresponding to the second collaborative feature expression; Based on the first classification result output by the first classification head and the third classification result output by the second classification head, calculate the global classification loss corresponding to each classification result to obtain the first loss function value corresponding to each classification result; and based on the second classification result output by the first classification head and the fourth classification result output by the second classification head, calculate the block classification loss corresponding to each classification result to obtain the second loss function value corresponding to each classification result; and based on the first distillation classification result output by the first classification head and the second distillation classification result output by the second classification head, calculate the distillation loss corresponding to each classification result to obtain the third loss function value corresponding to each classification result; and based on the fifth and sixth classification results, calculate the fusion feature classification loss corresponding to each classification result to obtain the fourth loss function value corresponding to each classification result. Backpropagation is performed based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value corresponding to each classification result to update the model parameters of the face liveness detection model; The first distillation classification result is obtained by inputting the first global feature vector and the first block feature vector into the first distillation classification head of the first classification head; the second distillation classification result is obtained by inputting the second global feature vector and the second block feature vector into the second distillation classification head of the second classification head.

8. The method according to claim 4, characterized in that, The step of inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: A third face image is generated based on the second face image, and the second face image, the second block image corresponding to the second face image, the third face image, and the third block image corresponding to the third face image are input into the face liveness detection model. The detection result is generated based on the classification result of the second classification head. The third face image is a visible light image of the same size as the second face image; the third segmented image is an image obtained by cropping the third face image. The step of inputting the face image to be detected into the face liveness detection model to obtain the detection result includes: The detection result is generated based on the third classification result and the fourth classification result.

9. A multimodal face liveness detection device, characterized in that, The device includes: The acquisition module is used to acquire the face image of the target object to be detected; the face image to be detected includes: visible light face image and infrared light face image; The detection module is used to input the face image to be detected into the face liveness detection model to obtain the detection result; The face liveness detection model includes: a first processing module for extracting feature information from visible light face images, and a second processing module for extracting feature information from infrared light face images; the face liveness detection model further includes: a multi-head classification module; the multi-head classification module is used to classify and predict the feature information extracted by the first processing module and the feature information extracted by the second processing module to obtain a face classification result; the face liveness detection model is used to generate the detection result based on the classification result output by the multi-head classification module; the detection result is used to characterize whether the target object is a real face.

10. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal face liveness detection method as described in any one of claims 1 to 8.

Images