Face recognition model training method for open scene and face recognition method

Abstract

The invention belongs to the technical field of computer vision, and discloses a face recognition model training method for an open scene and a face recognition method.According to the scheme, multiple modal images are used, and the modal images are segmented into image blocks; taking the feature vector corresponding to each image block as a node to connect adjacent nodes in the same image, establishing a cross-modal edge at the same position of different modals, establishing a time sequence edge at the same position of adjacent frames, and constructing a graph structure fusing spatial proximity, inter-modal consistency and time continuity; performing graph convolution operation based on the edge weight through a face recognition model to obtain feature representation of node update; a first loss function is obtained based on the updated feature representation and classification label similarity, a second loss function is obtained based on the multi-modal same position node initial feature vector distance, a third loss function is obtained based on the adjacent frame same position node initial feature vector distance, and a total loss function is constructed; and the trained face recognition model is high in recognition accuracy and robustness.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a face recognition model training method and a face recognition method for open scenes. Background Technology

[0002] In identity verification and recognition, it is necessary to acquire facial images and then use technologies such as computer vision and deep learning to extract facial features to determine identity. Current mainstream facial recognition methods use deep learning-based liveness detection models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). These models are trained by learning from a large amount of facial image data to obtain a facial recognition model for facial identification.

[0003] However, in real-world applications in open environments, environmental factors present significant challenges, including drastic changes in lighting, partial occlusion, pose diversity, and dynamic interference. Existing technologies generally rely on single-modal images for model training, leading to significant deficiencies in face recognition models in open scenarios: unstable feature extraction under low or high light conditions, a substantial drop in recognition accuracy due to occlusion or pose shifts, and insufficient ability to distinguish between live and malicious samples. This inability to effectively address the variability of open environments results in low accuracy and robustness of the models, failing to meet the practical requirements for high-reliability identity verification. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a face recognition model training method and a face recognition method for open scenarios.

[0005] According to one aspect of the embodiments of this application, a method for training a face recognition model for open scenarios is disclosed, the method comprising: Acquire multimodal images of a human face and perform preprocessing. The multimodal images include at least two modal images selected from multichannel images, depth images, and infrared images. Each preprocessed modal image is divided into a first number of image blocks, and features are extracted from each image block to generate an initial feature vector corresponding to each image block. Each initial feature vector is taken as a node, and the node is connected to multiple adjacent nodes in the same image, nodes at the same position in other modal images, and nodes at the same position in adjacent image frames to construct a graph structure. Calculate the edge weights of the connecting edges between nodes in the graph structure to obtain graph structure data with edge weights; A face recognition model is trained based on the graph structure data, and the face recognition model performs graph convolution operation based on the edge weights to obtain the updated feature representation of each node in the graph structure data. A first loss function is obtained based on the similarity between the feature representation and the classification label; a second loss function is obtained based on the distance between the initial feature vectors of nodes at the same position in the multimodal image; and a third loss function is obtained based on the distance between the initial feature vectors of nodes at the same position in adjacent image frames. A total loss function is then constructed based on the first loss function, the second loss function, and the third loss function. The model parameters of the face recognition model are iteratively updated based on the total loss function.

[0006] In some embodiments, constructing a total loss function based on the first loss function, the second loss function, and the third loss function includes: performing a weighted operation on the first loss function, the second loss function, and the third loss function to obtain the total loss function.

[0007] In some embodiments, obtaining the first loss function based on the similarity between the feature representation and the classification label includes: based on a relational expression The first loss function is obtained; where, For the first loss function, The feature is represented as follows: These are classification labels used to identify live samples and attack samples. For similarity calculation function, This represents the number of samples.

[0008] In some embodiments, obtaining the second loss function based on the distance between the initial feature vectors of nodes at the same location in the multimodal image includes: based on the relational expression The second loss function is obtained; where, For the second loss function, This is the initial feature vector of the multi-channel image. Let be the initial feature vector of the depth image. Let be the initial feature vector of the infrared image. It is an L2 norm.

[0009] In some embodiments, obtaining the third loss function based on the distance between the initial feature vectors of nodes at the same location in adjacent image frames includes: based on the relational expression The third loss function is obtained; where, For the third loss function, This is the initial feature vector of the current image frame. This is the initial feature vector of the previous image frame. It is an L2 norm.

[0010] In some embodiments, calculating the edge weights of the connecting edges between nodes in the graph structure to obtain graph structure data with edge weights includes: calculating the edge weights of the connecting edges between nodes in the graph structure using a bilinear attention mechanism to obtain graph structure data with edge weights. Performing graph convolution operations based on the edge weights to obtain updated feature representations of each node in the graph structure data includes: weighted aggregation of the feature representations of the target node's neighboring nodes based on the edge weights to obtain the updated feature representation of the target node, where the target node is any node in the graph structure data whose feature representation needs updating, and the neighboring nodes are nodes connected to the target node by edges.

[0011] In some embodiments, the step of performing graph convolution based on the edge weights to obtain the updated feature representation of each node in the graph structure data further includes: introducing a spatiotemporally gated recurrent unit, using the hidden state of the previous image frame as a temporal prior to capture fine-grained dynamic features, including micro-expression features and optical flow micro-motion features; and obtaining the updated feature representation of the target node based on the fine-grained dynamic features.

[0012] In some embodiments, the method further includes: calculating the motion trajectory and shape changes of pixels in a multimodal image sequence to capture micro-expression features of a face, wherein the multimodal image sequence is a sequence of multimodal images at multiple time points; calculating the phase local gradient information of the multimodal images to obtain illumination-invariant features that are insensitive to illumination changes; modeling long-term behavioral patterns based on a sliding window, wherein the long-term behavioral patterns include head posture, blinking frequency, and lip micro-movements; and inputting the micro-expression features, the illumination-invariant features, and the long-term behavioral patterns into the face recognition model to train the face recognition model.

[0013] In some embodiments, each modal image in the multimodal image is acquired based on a timestamp hard synchronization mechanism; preprocessing the multimodal image includes: adjusting the multimodal images to the same resolution; performing automatic exposure and white balance calibration on the multimodal images to unify the brightness and hue of the multimodal images; and performing distortion correction and pupil normalization processing on the multimodal images.

[0014] In some embodiments, the method further includes: constructing a support set and a query set, wherein the support set contains live samples and new attack samples, and the number of samples in the query set is greater than the number of samples in the support set; training the face recognition model using the support set, calculating the first loss function of the trained face recognition model on the support set, and updating the meta-parameters based on the first loss function; and using the trained face recognition model to participate in the next training until the training termination condition is met.

[0015] In some embodiments, the method further includes: summing the edge weights of the connecting edges between nodes in the multi-channel image to generate an attention heatmap, wherein the attention heatmap represents the degree of attention the face recognition model pays to different regions of the multi-channel image.

[0016] In some embodiments, the method further includes: reducing the dimensionality of the initial feature vector and training a decision tree using the reduced dimensionality initial feature vector, wherein the decision tree is used to explain the decision-making process of the face recognition model.

[0017] According to one aspect of the embodiments of this application, a face recognition method for open scenes is disclosed, comprising: acquiring a face image to be recognized, the face image including an RGB image and at least two modal images selected from a depth image and an infrared image; inputting the face image into a trained face recognition model to obtain a face recognition result, wherein the trained face recognition model is trained according to the above method.

[0018] The technical solutions provided by the embodiments of this application have at least the following beneficial effects: The scheme disclosed in this application uses at least two modal images to obtain facial feature information under different modalities. It segments each preprocessed modal image into image blocks and extracts features to generate initial feature vectors. Each image block is used as a node to connect adjacent nodes in the same image, establishes cross-modal connections at the same positions in different modalities, and establishes temporal connections at the same positions in adjacent frames, constructing a graph structure that integrates spatial proximity, intermodal consistency, and temporal continuity. By inputting the graph structure data with edge weights into the model, the face recognition model performs graph convolution operations based on the edge weights to obtain updated feature representations for nodes. Then, based on the similarity between the feature representations and classification labels, a first loss function is obtained; a second loss function is obtained based on the distance between the initial feature vectors of nodes at the same position in multiple modalities; and a third loss function is obtained based on the distance between the initial feature vectors of nodes at the same position in adjacent frames. A total loss function is then constructed. The first loss function optimizes classification performance, the second loss function constrains intermodal feature consistency to promote fusion, and the third loss function maintains temporal stability. Finally, the model parameters are iteratively updated based on the total loss function to train a face recognition model with high accuracy and robustness, capable of adapting to complex open scenarios. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the principles of this application.

[0020] Figure 1 A flowchart of a face recognition model training method according to an embodiment of this application is shown.

[0021] Figure 2A detailed flowchart of the image preprocessing steps according to an embodiment of this application is shown.

[0022] Figure 3 A detailed flowchart of the spatiotemporal dynamic feature extraction steps of an embodiment of this application is shown.

[0023] Figure 4 A detailed flowchart of an embodiment of this application for training a face recognition model based on a meta-learning strategy is shown.

[0024] Figure 5 A detailed flowchart of the training model interpretability steps in one embodiment of this application is shown.

[0025] Figure 6 A flowchart of a face recognition method according to an embodiment of this application is shown.

[0026] Figure 7 A block diagram of a computer device according to an embodiment of this application is shown.

[0027] Figure 8 A computer system architecture block diagram is shown for implementing some embodiments of this application.

[0028] The annotations in the attached figures are explained as follows: 700. Computer equipment; 701. Processor; 702. Memory; 800. Computer system; 801. CPU; 802. ROM; 803. RAM; 804. Bus; 805. I / O interface; 806. Input section; 807. Output section; 808. Storage section; 809. Communication section; 810. Driver; 811. Removable media. Detailed Implementation

[0029] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this application will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0030] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features.

[0031] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] In related technologies, face recognition based on a single-modality face image has poor accuracy and robustness.

[0034] Furthermore, existing face recognition methods mainly include deep learning-based liveness detection methods and methods based on traditional image processing. Deep learning-based liveness detection methods typically employ models such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), training them using large amounts of facial image data. However, these methods often suffer from overfitting when faced with novel attack types (such as deepfakes, 3D-printed masks, and high-fidelity silicone headgear), leading to decreased generalization performance. Overfitting can be assessed by calculating the model's cross-entropy loss function. To evaluate, among which It's a real label. It predicts probabilities. When a model overfits to the training data, its loss function is small, but it performs poorly on validation data. Furthermore, the "black box" nature of deep learning models makes their decision-making process difficult to interpret and unable to provide reliable evidence. For example, the output of a CNN model can be represented as... ,in It is the input image. It is a weight matrix. It is a bias vector. It is an activation function. This complex nonlinear function makes the model's decision-making process difficult to understand.

[0035] Traditional image processing-based liveness detection methods mainly include facial landmark matching and image quality assessment. Facial landmark matching typically uses point-to-point or point-to-line error metrics, such as mean squared error (MSE) or the distance from a point to a line. Image quality assessment may involve calculating metrics such as image sharpness and contrast. However, these methods often fail to accurately identify and judge liveness detection in complex scenes and with novel attack types. For example, when using MSE for landmark matching, the matching error can be expressed as... ,in These are the coordinates of the actual feature points. It predicts the coordinates of feature points. This refers to the number of feature points. This error metric may become inapplicable when the type of attack changes. Furthermore, traditional image processing methods have poor interpretability, making them difficult to meet the needs of regulators and users.

[0036] To address this, this application provides a face recognition model training method that uses at least two modal images to obtain facial feature information in different modalities. The method divides each preprocessed modal image into image blocks and extracts features to generate initial feature vectors. Each image block is used as a node to connect adjacent nodes in the same image, establishes cross-modal connections at the same positions in different modalities, and establishes temporal connections at the same positions in adjacent frames, constructing a graph structure that integrates spatial proximity, intermodal consistency, and temporal continuity. By inputting the graph structure data with edge weights into the model, the face recognition model performs graph convolution operations based on the edge weights to obtain updated feature representations for nodes. Then, based on the similarity between the feature representations and classification labels, a first loss function is obtained; the distance between the initial feature vectors of nodes at the same position in multiple modalities is obtained as a second loss function; and the distance between the initial feature vectors of nodes at the same position in adjacent frames is obtained as a third loss function, and a total loss function is constructed. The first loss function optimizes classification performance, the second loss function constrains intermodal feature consistency to promote fusion, and the third loss function maintains temporal stability. Finally, the model parameters are iteratively updated based on the total loss function to train a face recognition model with high accuracy and robustness, capable of adapting to complex open scenarios. Furthermore, by introducing meta-learning strategies and interpretable output schemes during model training, the face recognition model achieves strong generalization ability and interpretability.

[0037] First, we will explain some of the terms used in this application.

[0038] Multimodal images refer to facial image data containing multiple different types of information, which can include at least two modalities such as multichannel images (e.g., RGB images), depth images, and infrared images. Multichannel images provide rich color and texture information; depth images provide three-dimensional spatial structure information, helping to distinguish real faces from planar forgeries; and infrared images are insensitive to changes in lighting and can capture the unique thermal radiation characteristics of living individuals. By integrating this complementary information, the robustness of facial recognition models in complex open scenes can be enhanced.

[0039] Image patch: refers to a local region formed by segmenting the original image. In image processing, dividing an image into multiple image patches allows the model to focus on local details and extract features from different regions, thereby better understanding the local structure and texture information of the image.

[0040] Initial feature vectors: These are numerical vectors extracted from each image patch to represent the content of that patch. These vectors form the basis for subsequent graph structure construction and feature learning, containing visual feature information about the image patches.

[0041] Graph structure: Consists of nodes and connecting edges. Each image patch is considered a node, and the connecting edges between nodes represent the spatial, modal, or temporal relationships between image patches. This graph structure can effectively model complex relationships and support the propagation and aggregation of information between related nodes.

[0042] Edge weight: In graph data, edge weights are numerical values ​​representing the connection strength or importance between two nodes. In graph convolution operations, edge weights are used to weight and aggregate information from neighboring nodes, thus influencing the feature updates of the target node.

[0043] Graph convolution operations: a method for feature learning on graph-structured data. It updates the feature representation of the target node by aggregating feature information from the node's neighbors and weighting it with edge weights. This operation effectively captures local and global dependencies in graph structures, enhancing the expressive power of features.

[0044] Feature representation: This refers to the abstract numerical vector used to describe a face or its local regions after model processing. Updated feature representations are usually more discriminative than the initial feature vectors and can better reflect the essential features of a face.

[0045] The first loss function is a metric used to measure the classification performance of a model. It is calculated by comparing the similarity between the model's predictions and the true classification labels, aiming to guide the model to learn how to accurately distinguish between different facial identities or live / attack samples.

[0046] The second loss function is an indicator used to measure the consistency of features between nodes at the same location in multimodal images. It is obtained by calculating the distance between the initial feature vectors of corresponding regions in different modal images, aiming to encourage the model to learn shared feature representations across modalities, thereby enhancing the effect of modality fusion.

[0047] The third loss function is an index used to measure the stability of features at the same location in adjacent image frames. It is obtained by calculating the distance between the initial feature vectors of corresponding regions in consecutive time frames, aiming to maintain the continuity and stability of features in the time dimension, which helps to handle face changes in dynamic scenes.

[0048] Total loss function: A comprehensive loss metric that combines multiple loss functions. It guides the model training process by balancing the contributions of different loss terms to achieve multi-objective optimization and thus improve the overall performance of the model.

[0049] Model parameters: These refer to the adjustable weights and biases within the face recognition model being trained. These parameters are progressively optimized through iterative updates based on the total loss function, enabling the model to better learn from the input data and perform face recognition tasks.

[0050] Meta-parameters, also known as hyperparameters or meta-learning parameters, are the adjustable configurations and model parameters that control how a model quickly adapts to new attacks. They allow the model to rapidly adjust its parameters based on a small number of samples when facing unknown attacks (i.e., new attacks with zero or few samples), thus quickly adapting to new threats without having to train the entire model from scratch.

[0051] The implementation details of the technical solutions in the embodiments of this application are described in detail below.

[0052] Figure 1 A flowchart of a face recognition model training method according to an embodiment of this application is shown. (See attached document.) Figure 1 As shown, the face recognition model training method includes at least the following steps S101 to S107, which are described in detail below.

[0053] In step S101, a multimodal image of the face is acquired and preprocessed.

[0054] The multimodal image includes at least two modalities among multichannel images, depth images, and infrared images.

[0055] In some embodiments, a multimodal image may include a multichannel image and a depth image. In some embodiments, a multimodal image may include a multichannel image and an infrared image. In some embodiments, a multimodal image may include a depth image and an infrared image. In some embodiments, a multimodal image may further include a multichannel image, a depth image, and an infrared image.

[0056] Multi-channel images can be image data including red, green, and blue channels (RGB images), which reflect features such as skin tone, texture, and expression. Depth images are facial depth information acquired using techniques such as structured light or Time-of-Flight (ToF). This information provides the three-dimensional structure and contour of the face, helping to distinguish real faces from fake faces such as masks or screens. Infrared images capture infrared radiation information from the facial surface. This information is unaffected by lighting conditions and can reveal the temperature distribution of the facial surface, which is particularly important for detecting attacks using 3D-printed masks or silicone headgear.

[0057] In this multimodal image, each modal image is acquired using a timestamp-based hard synchronization mechanism. This ensures precise temporal alignment of the images, avoiding image deviations caused by asynchronous acquisition times and improving the accuracy of subsequent image processing and feature extraction.

[0058] In some embodiments, the timestamp hard synchronization mechanism requires a time synchronization error of less than 1 millisecond, which means that each modal image is captured at the same moment or within a very short time interval. Specifically, a precise clock synchronization protocol, such as IEEE 1588 (Precise Time Protocol, PTP) or NTP (Network Time Protocol), can be used during image acquisition, along with hardware-level time synchronization circuit design.

[0059] like Figure 2 As shown, multimodal image preprocessing may include the following steps S201 to S203, which are described in detail below.

[0060] In step S201, the multimodal images are adjusted to the same resolution.

[0061] In some embodiments, the multi-channel image, depth image, and infrared image are uniformly adjusted to a resolution of 256×256 to eliminate resolution differences between different modal images, thereby facilitating subsequent feature extraction and model training. Image scaling algorithms, such as bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation, can be used to enlarge or reduce the modal image to the target resolution.

[0062] In step S202, automatic exposure and white balance calibration are performed on the multimodal image to unify the brightness and hue of the multimodal image.

[0063] In some embodiments, automatic exposure dynamically adjusts gain or exposure time by analyzing the histogram or mean brightness of the modal images. White balance calibration estimates the color temperature of scene light sources and performs color correction using gray-world algorithms, perfect reflection algorithms, or learning-based algorithms. Alternatively, a deep learning-based approach can be used to train an end-to-end network that receives the original image and outputs an image after exposure and white balance calibration, thereby achieving uniformity in brightness and hue and ensuring consistent visual effects for multimodal images acquired under different lighting conditions.

[0064] In some embodiments, automatic exposure adjusts the camera's aperture, shutter speed, and ISO parameters to achieve moderate image brightness, avoiding overexposure or underexposure. White balance calibration adjusts the camera's color gain to ensure that images captured under different light sources have natural colors, avoiding color cast.

[0065] In step S203, distortion correction and pupil normalization are performed on the multimodal image.

[0066] In some embodiments, distortion correction is achieved through geometric transformations, such as perspective or radial transformations, to correct distorted images into orthographic images. Distortion correction can restore the true shape and structure of facial images, improving the accuracy of feature extraction and model training. Pupil normalization adjusts pupil size using image processing techniques, such as image scaling or interpolation, to bring pupils of different sizes to the same size, facilitating subsequent feature extraction and model training. This eliminates the impact of camera lens distortion and pupil size differences on image quality, ensuring image accuracy and consistency.

[0067] exist Figure 2In the illustrated embodiment, multimodal images are acquired using a timestamp-based hard synchronization mechanism, ensuring high temporal consistency between image frames from different modalities. This provides accurate raw data for subsequent feature extraction. The timestamp-based hard synchronization mechanism avoids information misalignment caused by time deviations, ensuring that the same location in different modal images truly corresponds to the physical entity at the same moment. Based on this, a series of standardized preprocessing operations are performed on the acquired multimodal images, including: adjusting all modal images to the same resolution, which unifies image size and pixel density, making different modal images comparable in spatial dimension; eliminating brightness differences and color casts that may occur under different modal sensors or lighting conditions through automatic exposure and white balance calibration, ensuring consistency in image brightness and hue, thereby reducing interference from lighting changes on facial feature extraction; and further correcting geometric distortion and standardizing facial pose and size through distortion correction and pupil normalization. After precise synchronization and standardized preprocessing, the image blocks of the multimodal images, after segmentation, can generate more accurate and robust initial feature vectors. When these initial feature vectors are constructed into graph-structured data, the connections between nodes in the graph structure more accurately reflect the true structure and dynamic information of the face because the quality and consistency of the image itself are guaranteed. This enables face recognition models to learn more effective and discriminative feature representations from high-quality input when performing graph convolution operations.

[0068] In step S102, the preprocessed modal images are divided into a first number of image blocks, and features are extracted from each image block to generate an initial feature vector corresponding to each image block.

[0069] In some embodiments, each preprocessed modal image is divided into 8×8 segments to obtain a series of 8×8 image blocks. These image blocks contain local information of the modal image, such as texture, color, and shape, providing a basis for subsequent feature extraction.

[0070] In some embodiments, feature extraction is performed on each image patch to generate a corresponding feature vector, i.e., an initial feature vector. Feature extraction can employ various methods, such as convolutional neural networks (CNNs) and deep learning models. In one embodiment of this application, a deep learning model is used to extract features from each image patch, generating a 256-dimensional feature vector. These feature vectors contain local information about the image patch, such as texture, color, and shape, providing a foundation for subsequent graph structure construction.

[0071] By mapping each image patch to a fixed-dimensional feature vector, subsequent graph convolution processing becomes easier. Furthermore, the 256-dimensional feature vector represents sufficient local information without making the model too large.

[0072] In step S103, each initial feature vector is used as a node, and the node is connected to multiple adjacent nodes in the same image, nodes at the same position in other modal images, and nodes at the same position in adjacent image frames to construct a graph structure.

[0073] That is, the connecting edges of a graph structure include intramodal edges, intermodal edges, and temporal edges.

[0074] In some embodiments, the construction of modal inner edges follows the 8-neighborhood connection rule, meaning each node is connected to its adjacent nodes in the same modal image in the horizontal, vertical, and diagonal directions. This connection method ensures the correlation between local features within the same modality, helping to capture the local structure and texture information of the image. Specifically, for a node in an image, its modal inner edges will be connected to its adjacent nodes above, below, left, right, and four diagonal directions, effectively capturing the relationships between local features.

[0075] The construction of intermodal edges employs a fully connected strategy, meaning each node is connected to nodes at the same location in different modal images. This connection method enables the fusion of features from different modalities, helping to capture complementary information about faces across different modalities. Specifically, for a node located in a multi-channel image, its intermodal edges will be connected to nodes at the same location in both the depth and infrared images. This fully connected strategy allows the model to integrate feature information from different modalities, improving the accuracy of face recognition.

[0076] The construction of temporal edges connects nodes at the same position in adjacent image frames, forming a spatiotemporal graph. This connection method considers feature changes over time, helping to capture the dynamic features of faces. Specifically, for a node in the current image frame, its temporal edges will connect to nodes at the same position in the previous and next frames. This temporal connection enables the model to capture feature changes of faces at different points in time, improving its ability to distinguish between live and non-live samples.

[0077] By constructing intra-modal edges, inter-modal edges, and temporal edges, a complex network structure, or graph structure, is formed, which can effectively capture fine-grained features and spatiotemporal dynamic changes in facial images. The number of these edges and the connection rules can be adjusted and optimized according to specific application scenarios and performance requirements. For example, the complexity of the network can be controlled by increasing or decreasing the number of edges, and the network structure can be changed by adjusting the connection rules, thereby affecting the system's performance and efficiency.

[0078] In step S104, the edge weights of the connecting edges between nodes in the graph structure are calculated to obtain graph structure data with edge weights.

[0079] In some embodiments, the edge weights of the connecting edges between nodes in the graph structure are calculated using a bilinear attention mechanism to obtain graph structure data with edge weights. Calculating edge weights using a bilinear attention mechanism can efficiently measure the similarity between nodes, achieving intramodal, intermodal, and temporal feature alignment, and providing more reliable weight allocation. The edge weights are updated in real time as the data changes, enabling the face recognition model to adapt to feature variations in different scenarios.

[0080] For example, the edge weights of the connecting edges between nodes in a graph structure are calculated based on the following relation: ;in, For edge weights, and Let be a learnable weight matrix, and let vi and vj be two nodes connected by an edge. It is the node feature dimension.

[0081] In step S105, a face recognition model is trained based on graph structure data. The face recognition model performs graph convolution operation based on edge weights to obtain the updated feature representation of each node in the graph structure data.

[0082] In some embodiments, the feature representations of the target node's neighboring nodes are weighted and aggregated based on edge weights to obtain the updated feature representation of the target node. The target node is any node in the graph structure data whose feature representation needs to be updated, and the neighboring nodes are nodes that have connecting edges with the target node.

[0083] We perform weighted aggregation of the feature representations of neighboring nodes based on edge weights to ensure that the contribution is dynamically adjusted according to the association strength, thereby obtaining a more accurate feature representation of the target node.

[0084] In some embodiments, a message passing mechanism is used to aggregate information from neighboring nodes and perform graph convolution operations. This method can capture the relationships between nodes and extract local features of the image. Specifically, each node... Features By aggregating its set of neighboring nodes All nodes Features This is for updating. During the aggregation process, the edge weights... Used to measure nodes With nodes The strength of the correlation between features. Through this graph convolution operation, the model can capture the relationships between local features in an image, thereby extracting more discriminative feature representations.

[0085] For example, graph convolution operations are performed based on the following relation: ;in, For nodes In the The updated feature representation of the layer, For nodes In the Features of the layer It is a node The set of neighboring nodes, It is the first The weight matrix of the layer, It is an activation function.

[0086] In some embodiments, in addition to calculating the edge weights of the connecting edges between nodes in the graph structure and performing graph convolution operations based on the edge weights to obtain the updated feature representations of each node in the graph structure data, a spatiotemporal gated recurrent unit is also introduced. The hidden state of the previous image frame is used as a temporal prior to capture fine-grained dynamic features, including micro-expression features and optical flow micro-motion features. Based on the fine-grained dynamic features, the updated feature representation of the target node is obtained.

[0087] The spatiotemporally gated recurrent unit (GRU) is a neural network module used to process data with both spatial and temporal dependencies. It combines the spatial information processing capabilities of graph convolutional networks (GRNs) with the temporal information processing capabilities of GRUs. Specifically, the spatiotemporally gated GRU selectively memorizes and forgets historical information through internal gating mechanisms (such as update and reset gates), effectively capturing the dynamic changes of node features over time. This can be achieved by using the output of graph convolution operations as input to the GRU, or by designing a composite unit that integrates graph convolution and GRU gating mechanisms. For example, by introducing graph convolution operations into the linear transformation part of the GRU, each node, when updating its hidden state, considers not only its own temporal evolution but also the spatial influence of its neighboring nodes.

[0088] By using the hidden state of the previous image frame as a temporal prior, the spatiotemporally gated recurrent unit can capture the dynamic changes of the face at different time points, thereby improving the model's ability to distinguish between live and non-live samples.

[0089] Fine-grained dynamic features refer to minute, rapid, and often imperceptible movement patterns within the facial region. These features reflect the physiological activities of the face, such as micro-expressions and subtle eye movements, and are crucial for distinguishing live faces from various attacks (such as photos, video playback, and 3D masks). Micro-expression features are involuntary facial expressions lasting extremely short periods (usually less than 0.5 seconds), revealing an individual's true inner emotions. In face recognition and liveness detection, micro-expressions are an important indicator of liveness. Optical flow micro-motion features specifically refer to extremely minute, pixel-level movements within the facial region, such as skin tremors and subtle eye movements. These micro-movements are inherent properties of live faces.

[0090] Based on fine-grained dynamic features, the updated feature representation of the target node is obtained. When updating node features in graph convolution operations, in addition to considering spatial neighborhood information, the captured fine-grained dynamic features are also incorporated into the feature representation of the node. This significantly improves the robustness and accuracy of the face recognition model in open scenarios, especially when facing various dynamic attacks.

[0091] In step S106, a first loss function is obtained based on the similarity between the feature representation and the classification label, a second loss function is obtained based on the distance between the initial feature vectors of nodes at the same position in the multimodal image, and a third loss function is obtained based on the distance between the initial feature vectors of nodes at the same position in adjacent image frames. A total loss function is then constructed based on the first loss function, the second loss function, and the third loss function.

[0092] In some embodiments, a first loss function is obtained based on the similarity between the feature representation and the classification label, including: based on a relational expression. The first loss function is obtained; where, For the first loss function, For feature representation, These are classification labels used to identify live samples and attack samples. For similarity calculation function, This represents the number of samples.

[0093] The similarity calculation function is used to calculate the similarity between the feature representation and the classification label, and the classification is optimized by negative log-likelihood loss, which helps to accurately distinguish between live samples and attack samples.

[0094] Understandably, the similarity between a feature representation and a classification label refers to the degree of similarity between the feature representation and the feature vector of a certain category. This similarity reflects the degree of similarity between the feature representation and the category.

[0095] The second loss function, also known as the cross-modal consistency loss, is used to reduce intermodal differences and ensure consistency between images of different modalities. It is achieved by calculating the distance between the initial feature vectors of the multi-channel image, the initial feature vector of the depth image, and the initial feature vector of the infrared image.

[0096] In some embodiments, a second loss function is obtained based on the distance between the initial feature vectors of nodes at the same location in the multimodal image, the distance between the initial feature vector of the multi-channel image and the initial feature vector of the depth image, and the distance between the initial feature vector of the multi-channel image and the initial feature vector of the infrared image. These two distances are then added together to obtain the final second loss function.

[0097] In some embodiments, based on relational The second loss function is obtained; where, For the second loss function, This is the initial feature vector of the multi-channel image. This is the initial feature vector of the depth image. This is the initial feature vector of the infrared image. It is an L2 norm.

[0098] The third loss function, also known as the temporal consistency loss, is used to ensure consistency between adjacent image frames. It is achieved by calculating the distance between the initial feature vectors of adjacent image frames. Specifically, the third loss function calculates the distance between the initial feature vector of the current image frame and the initial feature vector of the previous image frame, and uses this distance as the temporal consistency loss.

[0099] In some embodiments, based on relational The third loss function is obtained; where, For the third loss function, This is the initial feature vector of the current image frame. This is the initial feature vector of the previous image frame. It is an L2 norm.

[0100] Finally, these three loss functions are combined into a total loss function. In some embodiments, constructing the total loss function based on the first loss function, the second loss function, and the third loss function includes: weighting the first loss function, the second loss function, and the third loss function to obtain the total loss function.

[0101] By employing weighted operations, adjustable weights can be assigned to each loss, allowing for flexible control over the relative contributions of different losses during model training. This ensures that classification loss, modality consistency loss, and temporal consistency loss work synergistically during optimization, preventing a single loss from dominating training and causing the model to ignore crucial information such as multimodal features or temporal dynamics. This improves the stability and efficiency of model training, ultimately enhancing the robustness and accuracy of the face recognition model in open scenarios.

[0102] In some embodiments, based on relational Calculate the total loss function; where, For the total loss function, For the first loss function, For the second loss function, For the third loss function, , This is a hyperparameter and can be set based on experience.

[0103] In step S107, the model parameters of the face recognition model are iteratively updated based on the total loss function.

[0104] Optimization algorithms such as gradient descent can be used to gradually adjust the weights and biases within the model based on the gradient calculated from the total loss function. This iterative process continues until the model's performance on the validation set reaches expectations, or a preset training termination condition is met.

[0105] exist Figure 1 In the illustrated embodiment, acquiring multiple modal images, such as multi-channel images, depth images, and infrared images, provides richer and more complementary data for training the face recognition model. Images are segmented into image patches and graph-structured data is constructed, integrating spatial proximity, intermodal consistency, and temporal continuity into a unified graph structure. Simultaneously, a multi-loss function collaborative optimization mechanism is introduced. The first loss function ensures accurate classification of the liveness / attack category by the face recognition model. The second loss function, by constraining the distance between the initial feature vectors of nodes at the same location in different modal images, prompts the face recognition model to learn consistent feature representations across modalities, effectively promoting the deep fusion of multimodal information. The third loss function, by constraining the distance between the initial feature vectors of nodes at the same location in adjacent image frames, ensures the stability of features in the temporal dimension, helping to handle subtle dynamic changes in the face. During training, the face recognition model can simultaneously consider classification performance, modal fusion effect, and temporal stability, thereby training a face recognition model with stronger generalization ability and robustness in open scenarios.

[0106] In some embodiments, a series of physical feature extraction methods parallel to the initial feature vector generation are used to capture spatiotemporal dynamic features, and the captured spatiotemporal dynamic features are used for subsequent auxiliary supervision and feature fusion to enhance the robustness and interpretability of the face recognition model.

[0107] like Figure 3 As shown, the spatiotemporal dynamic feature extraction steps may include the following steps S301 to S304, which are described in detail below.

[0108] In step S301, the motion trajectory and shape change of pixels in the multimodal image sequence are calculated to capture the micro-expression features of the face. The multimodal image sequence is a sequence of multimodal images at multiple time points.

[0109] In some embodiments, TV-L1 optical flow and 3D-HOG features are employed to capture micro-expression features of the face by calculating the motion trajectory and local shape changes of pixels in a multimodal image sequence. These micro-expression features can provide information on the dynamic changes of the face, enhancing the system's ability to recognize various types of attacks.

[0110] In step S302, the phase local gradient information of the multimodal image is calculated to obtain illumination-invariant features that are insensitive to illumination changes.

[0111] In some embodiments, a PLGF (Phase Local Gradient Filter) branch is introduced to extract illumination-invariant features by calculating the phase local gradient information of multimodal images. These illumination-invariant features can eliminate the impact of illumination changes on feature extraction and model training, thereby improving the robustness of the face recognition model.

[0112] In step S303, long-term behavioral patterns are modeled based on a sliding window. These long-term behavioral patterns include head posture, blinking frequency, and lip micro-movements.

[0113] In some embodiments, an RNN-Transducer is used to model long-term behavioral patterns such as head pose, blink frequency, and lip micromovements over a 5-second sliding window. These long-term behavioral patterns provide continuous behavioral information about the face, enhancing the face recognition model's ability to identify various attack types.

[0114] In step S304, micro-expression features, illumination-invariant features, and long-term behavioral patterns are input into the face recognition model to train the face recognition model.

[0115] Specifically, micro-expression features, illumination-invariant features, and long-term behavioral patterns can be used as additional input channels or feature vectors, concatenated with the initial feature vectors extracted from multimodal images to form a longer, comprehensive feature vector. This vector is then fed into the fully connected layer or subsequent graph convolutional layer of the face recognition model to be trained. Alternatively, a multi-branch network structure can be designed, with each branch responsible for processing a specific type of feature. The outputs of these branches are then fused together in the later stages of the network before being fed into the final classifier.

[0116] exist Figure 3 In the illustrated embodiment, the introduction of spatiotemporal dynamic features significantly enhances the face recognition model's capabilities in dynamic micro-expressions, illumination robustness, and long-term behavior. Specifically, recognizing subtle facial fluctuations, such as muscle contraction or relaxation, helps distinguish between genuine faces and static attacks; extracting illumination-invariant features avoids recognition errors caused by changes in ambient lighting, improving the face recognition model's adaptability in open scenes; and modeling long-term behavioral patterns helps identify consistent long-term behavior, preventing deception based on transient attacks.

[0117] The face recognition model trained through the aforementioned embodiments possesses powerful fine-grained feature representation capabilities. However, when faced with zero-shot attacks originating from novel attack types that are not present in the training data, the model's decision boundary may fail. To address this issue, some embodiments of this application further introduce an interpretable meta-learning framework (MetaFAS).

[0118] like Figure 4As shown, training a face recognition model based on a meta-learning strategy may include the following steps S401 to S403, which are described in detail below.

[0119] In step S401, a support set and a query set are constructed. The support set contains live samples and new attack samples, and the number of samples in the query set is greater than the number of samples in the support set.

[0120] In other words, the support set contains a small number of live samples and new attack samples for rapid adaptation of meta-parameters. For example, the support set contains 4 live samples and 4 new attack samples. The query set contains a large number of samples for evaluating the generalization ability of the face recognition model. For example, the query set contains 16 samples.

[0121] The liveness samples were selected from facial images of normal users, representing genuine facial features. The new attack samples included facial images of various novel attack types, such as deepfakes, 3D-printed masks, and high-fidelity silicone headgear.

[0122] In step S402, a face recognition model is trained using the support set, and the first loss function of the trained face recognition model on the support set is calculated. The meta-parameters are then updated based on the first loss function.

[0123] By updating through inner and outer loops, it can quickly adapt to newly emerging attack types. For example, the inner loop uses a 5-step SGD process to update meta-parameters. Step length It's worth learning. The outer loop uses Adam to update meta-parameters. The update of the meta-learning strategy can be achieved using the following formula: ;in, These are meta-parameters. These are the updated meta-parameters. It's the learning rate. It is the loss function on the support set.

[0124] This two-layer optimization mechanism balances the face recognition model's rapid adaptability on the support set and its generalization ability on the query set. The inner loop's SGD update aims to minimize the loss on the current task's support set, enabling rapid updates of meta-parameters to adapt to new attack types. The outer loop's Adam update optimizes the model's initial parameters, ensuring excellent performance on the query set after a few updates in the inner loop. This approach ensures the model acquires good initialization capabilities, enabling rapid generalization on a small number of samples, rather than simply overfitting to a limited number of samples from new attacks, thus effectively balancing adaptation speed and generalization performance.

[0125] loss function Various forms can be used, such as cross-entropy loss and mean squared error loss. Figure 4 In the illustrated embodiment, AM-Softmax loss is used as... This is to distinguish between live samples and attack samples.

[0126] In some embodiments, only the γ and β of the last two +BatchNorm layers are updated, reducing the number of parameters by more than 90% and improving the rapid adaptability of the face recognition model.

[0127] In step S403, the trained face recognition model is used to participate in the next training until the training termination condition is met.

[0128] Training termination conditions can include reaching the maximum number of iterations, convergence of meta-parameters, and the face recognition model no longer showing improvement in performance on the validation set. Through this iterative training, the face recognition model gradually strengthens its generalization ability, ensuring high recognition accuracy when facing constantly evolving new attacks.

[0129] exist Figure 4 In the illustrated embodiment, based on the fine-grained, cross-modal aligned spatiotemporal dynamic features provided in the aforementioned embodiment, the meta-learning process can perform rapid and effective decision boundary adjustment in a high-dimensional, information-rich feature space. This enables the face recognition model to keenly capture subtle anomalies in local details of new attack patterns, quickly adapt to emerging attack types, and improve the generalization ability of the face recognition model.

[0130] In some embodiments, attention heatmaps and decision path visualization methods are used to improve the interpretability of face recognition models.

[0131] like Figure 5 As shown, the training model interpretability steps may include the following steps S501 to S503, which are described in detail below.

[0132] In step S501, the edge weights of the connecting edges between nodes in the multi-channel image are summed to generate an attention heatmap, which represents the degree of attention the face recognition model pays to different regions of the multi-channel image.

[0133] In some embodiments, the attention heatmap is generated by summing the edge weights of the connections between nodes in the multi-channel image. The resulting 256×256 saliency map (i.e., attention heatmap) visually demonstrates the degree of attention the face recognition model pays to different regions in a multi-channel image. It highlights the key regions that the face recognition model focuses on during the decision-making process, helping users and regulators understand the decision-making basis of the face recognition model.

[0134] In step S502, the initial feature vector is reduced in dimensionality, and a decision tree is trained using the reduced initial feature vector. The decision tree is used to explain the decision-making process of the face recognition model.

[0135] In some embodiments, decision path visualization is achieved by training an interpretable decision tree. By reducing the initial feature vectors to 32-d and then training the interpretable decision tree, the output IF-THEN-ELSE rules (depth ≤ 6) explain the decision-making process of the face recognition model. These rules are presented in IF-THEN-ELSE form, allowing users and regulators to intuitively understand the decision logic of the face recognition model.

[0136] In step S503, pseudo-labels are generated through the teacher network, and the student network is iteratively refined through consistency regularization and entropy minimization.

[0137] The teacher network is a large model trained on a massive dataset, capable of generating high-quality pseudo-labels. The student network, on the other hand, is a small model trained on a small number of samples, improving its accuracy by learning the pseudo-labels generated by the teacher network. Consistency regularization and entropy minimization help the student network learn more robust and generalized feature representations.

[0138] exist Figure 5 In the illustrated embodiment, the combination of attention heatmap, decision path visualization, and pseudo-label self-correction not only improves the interpretability of the face recognition model but also enhances its accuracy, resulting in higher credibility and reliability.

[0139] The following example illustrates the deployment and maintenance process of the applicant's face recognition model.

[0140] On mobile devices, high-performance processors are deployed, leveraging their integrated neural network processors for rapid inference. These processors can perform neural network calculations with 8-bit integer precision, significantly reducing computational complexity and power consumption while maintaining high inference accuracy. This deployment strategy enables fast face fraud detection on mobile devices, meeting real-time requirements.

[0141] To improve the model's adaptability, node features from the most recent 16 frames are cached on the mobile device. These node features are used for local meta-learning fine-tuning, enabling the model to quickly adjust to the user's specific situation, thereby improving the model's accuracy and robustness. Local meta-learning fine-tuning uses cached features to fine-tune the model by running a lightweight meta-learning algorithm on the mobile device, allowing the model to better adapt to the user's specific characteristics and environmental changes.

[0142] Within the edgebox, a teacher network is deployed, responsible for model distillation and anomaly sample backpropagation. Model distillation is a technique that transfers knowledge from a large model to a smaller model, training a small model to mimic the output of the large model, thereby reducing model size and computational requirements while maintaining high inference accuracy. Anomaly sample backpropagation refers to transmitting anomaly samples (such as unknown attack types) detected by the edgebox back to the cloud for further analysis and processing, thereby improving the overall security of the system.

[0143] The specific implementation of model distillation is achieved by optimizing the following formula: ;in, These are parameters of the student network. These are parameters of the teacher network. This is the distillation loss function. The distillation loss function measures the difference between the student network's output and the output of the large model. By minimizing this loss function, the student network can learn the knowledge of the large model, thereby improving its inference accuracy.

[0144] High-performance computing clusters are deployed in the cloud for training global meta-parameters. Training global meta-parameters involves optimizing the meta-parameters within the meta-learning framework to improve the model's generalization and adaptability. This training is typically performed on large-scale datasets, fully utilizing cloud computing resources to enhance training efficiency and model performance.

[0145] To meet regulatory requirements, a compliance audit interface is provided, outputting JSON logs containing information such as decision tree path, attention heatmap, model version, and hash value. These logs record the model's decision-making process and related information, making the system's decision-making process transparent and traceable, thus meeting the regulatory agency's requirements for system compliance.

[0146] By coordinating deployment and operation across mobile devices, edge boxes, and the cloud, the facial recognition model in this application achieves efficient deployment and operation while maintaining high performance, meeting the stringent requirements of industrial applications for real-time performance, interpretability, and compliance. This layered architecture effectively integrates computing resources from the cloud, edge, and terminal, improving deployment efficiency and operational capabilities.

[0147] Figure 6 A flowchart of a face recognition method according to an embodiment of this application is shown. (See attached document.) Figure 6 As shown, the face recognition method includes at least the following steps S601 to S602, which are described in detail below.

[0148] In step S601, a face image to be identified is obtained. The face image includes an RGB image and at least two modal images from a depth image and an infrared image.

[0149] In step S602, the face image is input into the trained face recognition model to obtain the face recognition result.

[0150] In summary, this application provides richer and more complementary data for training face recognition models by acquiring multi-channel images, depth images, and infrared images, among other modalities. It segments images into image patches and constructs graph-structured data, integrating spatial proximity, intermodal consistency, and temporal continuity into a unified graph structure. Simultaneously, it introduces a multi-loss function collaborative optimization mechanism. The first loss function ensures accurate classification of liveness / attack categories by the face recognition model. The second loss function, by constraining the distance between the initial feature vectors of nodes at the same location in different modal images, encourages the face recognition model to learn consistent feature representations across modalities, effectively promoting the deep fusion of multimodal information. The third loss function, by constraining the distance between the initial feature vectors of nodes at the same location in adjacent image frames, ensures the stability of features in the temporal dimension, helping to handle subtle dynamic changes in the face. During training, the face recognition model can simultaneously consider classification performance, modal fusion effect, and temporal stability, thereby training a face recognition model with stronger generalization ability and robustness in open scenarios. Furthermore, by introducing meta-learning strategies and interpretable output schemes during model training, the face recognition model achieves strong generalization ability and interpretability.

[0151] Experiments show that the applicant's face recognition model reduces the false acceptance rate (EER) to below 10% in zero-shot attack scenarios, a reduction of 53%. This means that even with attack types not present in the training data, the model can still effectively identify attack behaviors, thereby reducing the probability of falsely accepting attack samples and significantly improving the model's robustness against new attack types. In cross-database tests using samples from different datasets, the half total error rate (HTER) is reduced to below 5%, a reduction of 36%, improving the model's accuracy when processing data from different sources and thus enhancing its reliability in practical applications. Achieving a single inference time of less than 200 milliseconds on mobile devices enables rapid face recognition on mobile devices, meeting real-time requirements for real-time identity verification in mobile scenarios such as mobile banking and mobile payments. Furthermore, no user interaction is required during use, improving the user experience and making it more convenient to apply in various scenarios. Ultimately, the facial recognition model achieved remarkable results in terms of robustness, generalization ability, and interpretability, meeting the stringent requirements of industrial-grade deployment and possessing broad application prospects. It can be used in various application scenarios such as financial risk control, smart access, and trusted identity in government affairs.

[0152] See next. Figure 7This embodiment provides a computer device 700, which includes one or more processors 701 and a memory 702. The memory 702 is used to store one or more programs. When one or more programs are executed by one or more processors 701, the computer device 700 implements the face recognition model training method of this application.

[0153] Figure 8 The diagram shows a computer system architecture block diagram for implementing some embodiments of this application. It should be noted that... Figure 8 The computer system shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0154] like Figure 8 As shown, the computer system 800 includes a CPU (Central Processing Unit) 801, which can perform various appropriate actions and processes based on programs stored in ROM (Read-Only Memory) 802 or programs loaded from storage portion 808 into RAM (Random Access Memory) 803, such as executing the face recognition model training method described in the above embodiment. The RAM 803 also stores various programs and data required for system operation. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An I / O (Input / Output) interface 805 is also connected to the bus 804.

[0155] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0156] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing all or some of the steps shown in the flowcharts of the face recognition model training method. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs various functions defined in the system of this application.

[0157] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0159] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0160] In another aspect, this application also provides a computer-readable medium, which may be included in the computer device described in the above embodiments; or it may exist independently and not assembled into the computer device. The computer-readable medium carries one or more programs that, when executed by the computer device, cause the computer device to implement the methods described in the above embodiments.

[0161] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0162] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0163] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

Claims

1. A method for training a face recognition model for open scenarios, characterized in that, The method comprises: obtaining a multi-modal image of a face and preprocessing the multi-modal image, the multi-modal image comprising at least two modal images of a multi-channel image, a depth image and an infrared image; dividing each of the preprocessed modal images into a first number of image blocks, and extracting features of each of the image blocks to generate an initial feature vector corresponding to each of the image blocks; taking each initial feature vector as a node, connecting the node with a plurality of adjacent nodes in the same image, nodes at the same position in other modal images and nodes at the same position in adjacent image frames to construct a graph structure; calculating edge weights of connecting edges between nodes in the graph structure to obtain graph structure data with edge weights; training a face recognition model based on the graph structure data, performing graph convolution operation based on the edge weights by the face recognition model to obtain updated feature representations of each node in the graph structure data; obtaining a first loss function based on the similarity between the feature representations and classification labels, a second loss function based on the distance between initial feature vectors of nodes at the same position in the multi-modal image, and a third loss function based on the distance between initial feature vectors of nodes at the same position in adjacent image frames, and constructing a total loss function based on the first loss function, the second loss function and the third loss function; iteratively updating model parameters of the face recognition model based on the total loss function.

2. The method of claim 1, wherein, The method comprises: performing weighted operation on the first loss function, the second loss function and the third loss function to obtain the total loss function.

3. The method of claim 1, wherein: the first loss function is obtained based on the similarity between the feature representations and classification labels, and / or based on the relation , a first loss function is obtained; wherein, is a first loss function, is the feature representation, is a classification label identifying live samples and attack samples, is a similarity computation function, is a number of samples; the second loss function is obtained based on the distance between initial feature vectors of nodes at the same position in the multi-modal image, and / or the third loss function is obtained based on the distance between initial feature vectors of nodes at the same position in adjacent image frames. based on the relation , a second loss function is obtained; wherein, is a second loss function, is an initial feature vector of the multi-channel image, is an initial feature vector of the depth image, is an initial feature vector of the infrared image, is an L2 norm; 4. The method of claim 1, wherein: the edge weights of connecting edges between nodes in the graph structure are calculated to obtain graph structure data with edge weights, comprising: based on the relationship , a third loss function is obtained; wherein, is a third loss function, is an initial feature vector of a current image frame, is an initial feature vector of a previous image frame, is an L2 norm. the edge weights of connecting edges between nodes in the graph structure are calculated by a bilinear attention mechanism to obtain graph structure data with edge weights; the graph convolution operation based on the edge weights is performed to obtain updated feature representations of each node in the graph structure data, comprising: the feature representations of neighbor nodes of a target node are weighted and aggregated based on the edge weights to obtain updated feature representations of the target node, the target node being any node with feature representations to be updated in the graph structure data, and the neighbor nodes being nodes having connecting edges with the target node. the graph convolution operation based on the edge weights is performed to obtain updated feature representations of each node in the graph structure data, further comprising: ​ 5. The method of claim 4, wherein, ​ The spatio-temporal gating recurrent unit is introduced, and the hidden state of the previous image frame is taken as a temporal prior to capture fine-grained dynamic features, including micro-expression features and optical flow micro-motion features. Based on the fine-grained dynamic features, the updated feature representation of the target node is obtained.

6. The method of claim 1, wherein, Further comprising: The motion trajectory and shape change of the pixel points in the multi-modal image sequence are calculated to capture the micro-expression features of the face, and the multi-modal image sequence is a sequence composed of multi-modal images at multiple time points. The phase local gradient information of the multi-modal image is calculated to obtain illumination-invariant features that are not sensitive to illumination changes. The long-term behavior patterns, including head posture, blinking frequency and lip micro-motion, are modeled based on a sliding window. The micro-expression features, the illumination-invariant features and the long-term behavior patterns are input into the face recognition model to train the face recognition model.

7. The method of claim 1, wherein, Each modality image in the multi-modal image is collected based on a timestamp hard synchronization mechanism. The preprocessing of the multi-modal image includes: The multi-modal image is adjusted to the same resolution; The multi-modal image is subjected to automatic exposure and white balance calibration to unify the brightness and hue of the multi-modal image; The multi-modal image is subjected to distortion correction and pupil normalization processing.

8. The method according to any one of claims 1 to 7, characterized in that, Further comprising: A support set and a query set are constructed, the support set containing live samples and new attack samples, and the number of samples in the query set is greater than that in the support set; The face recognition model is trained using the support set, and the first loss function of the trained face recognition model on the support set is calculated, and the meta-parameter is updated based on the first loss function; The trained face recognition model is used for the next training until the training termination condition is met.

9. The method of claim 8, wherein, Further comprising: The edge weights of the connection edges between the nodes in the multi-channel image are summed to generate an attention heat map, which represents the attention degree of the face recognition model to different regions of the multi-channel image. And / or The initial feature vector is reduced in dimension, and the reduced initial feature vector is used to train a decision tree, which is used to explain the decision-making process of the face recognition model.

10. An open scene oriented face recognition method, characterized in that, It includes: Obtain a face image to be identified, which includes an RGB image and at least two modality images of a depth image and an infrared image; Input the face image into the trained face recognition model to obtain a face recognition result, wherein the trained face recognition model is trained according to the method of any one of claims 1 to 9.

Images