Multi-modal face data feature fusion method and system

Abstract

The invention discloses a multi-modal face data feature fusion method and system, relates to the field of image processing, abandons a mode of directly and simply fusing original features, and firstly introduces a cross-modal recalibration mechanism. According to the mechanism, before formal fusion, information interaction and mutual calibration are carried out on the characteristics of the visible light mode and the near-infrared mode, the intensities and expressions of the characteristics of different modes are adjusted and balanced, and information mismatch between the modes is relieved fundamentally. On the basis, a fusion weight is adaptively distributed to each re-calibrated modal feature according to the characteristics of the current input data through a dynamic gating fusion network. In this way, the fusion process can intelligently judge and improve the contribution degree of more advantageous modes (such as near-infrared features under dark light) in a specific scene, continuous dominance of a single mode is effectively avoided, and therefore full utilization and efficient integration of multi-mode information are achieved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to a method and system for fusing multimodal facial data features. Background Technology

[0002] With the widespread application of facial recognition technology in fields such as security, finance, and health management, the limitations of single-modal facial recognition solutions in complex environments are becoming increasingly apparent. For example, recognition systems relying solely on visible light (RGB) images experience a sharp decline in performance under poor lighting conditions (such as at night or in strong light). To improve recognition accuracy and robustness, the industry has begun to shift towards multimodal facial recognition technology that integrates visible light and near-infrared (NIR) images, as near-infrared imaging can effectively overcome the adverse effects of changes in lighting conditions.

[0003] However, existing multimodal fusion solutions generally face the core technical challenge of modal information imbalance and dominance. This imbalance stems from two aspects: first, the differences in the physical characteristics of different sensors, such as the inherent gap in resolution and imaging quality between visible light cameras and near-infrared sensors; second, the differences in the capabilities of feature extraction networks. Visible light image processing typically employs powerful backbone networks pre-trained on massive amounts of data, while the near-infrared branch may use a lighter network due to a lack of large-scale pre-training data. These differences result in features extracted from the visible light modality having far greater information content and discriminative power than those from the near-infrared modality during feature fusion. If only simple feature concatenation or weighting is performed, visible light features will dominate, preventing the model from fully learning and utilizing the unique advantages of near-infrared features in specific scenarios, thus undermining the original intention of multimodal fusion.

[0004] Therefore, we aim to propose a novel feature fusion method. Summary of the Invention

[0005] To address the aforementioned modal imbalance and dominance issues, according to one aspect of this application, a multimodal face data feature fusion method is provided, comprising: acquiring visible light face images and near-infrared face images.

[0006] Parallel multimodal feature extraction is performed on visible light face images and near-infrared face images to obtain the original visible light features and the original near-infrared features of the face.

[0007] Cross-modal recalibration was performed on the original visible light features and original near-infrared features of the face to obtain the recalibrated visible light features and recalibrated near-infrared features of the face.

[0008] Dynamic gating feature fusion is performed on the recalibrated visible light features and recalibrated near-infrared features of the face to obtain multimodal fusion features of the face.

[0009] According to another aspect of this application, a multimodal face data feature fusion system is provided, which includes: a multimodal face image acquisition module for acquiring visible light face images and near-infrared face images.

[0010] The face image multimodal feature extraction module is used to perform parallel multimodal feature extraction on visible light face images and near-infrared face images to obtain the original visible light features and the original near-infrared features of the face.

[0011] The feature recalibration module is used to perform cross-modal recalibration of the original visible light features and the original near-infrared features of the face to obtain the recalibrated visible light features and the recalibrated near-infrared features of the face.

[0012] The multimodal fusion module is used to dynamically gated feature fusion of the recalibrated visible light features and the recalibrated near-infrared features of the face to obtain multimodal fused features of the face.

[0013] Compared with existing technologies, this application provides a multimodal face data feature fusion method and system, which proposes a two-stage strategy of recalibration followed by fusion to address the strong modality dominance problem caused by the imbalance of feature representation capabilities in multimodal face recognition. This scheme abandons the simple fusion approach of directly fusing the original features, first introducing a cross-modal recalibration mechanism. Before formal fusion, this mechanism allows features from visible light and near-infrared modalities to interact and mutually calibrate, aiming to adjust and balance the intensity and expression of different modal features, fundamentally alleviating the information asymmetry between modalities. Based on this, a dynamic gated fusion network adaptively assigns fusion weights to each recalibrated modal feature according to the characteristics of the current input data. This enables the fusion process to intelligently determine and enhance the contribution of modalities that are more advantageous in specific scenarios (such as near-infrared features in low light), effectively avoiding the continuous dominance of a single modality, thereby achieving full utilization and efficient integration of multimodal information. Attached Figure Description

[0014] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0015] Figure 1 This is a flowchart of a multimodal face data feature fusion method according to an embodiment of this application.

[0016] Figure 2This is a schematic diagram of the data flow of the multimodal face data feature fusion method according to an embodiment of this application.

[0017] Figure 3 This is a flowchart of step S3 in the multimodal face data feature fusion method according to an embodiment of this application.

[0018] Figure 4 This is a flowchart of step S4 in the multimodal face data feature fusion method according to an embodiment of this application.

[0019] Figure 5 This is a block diagram of a multimodal face data feature fusion system according to an embodiment of this application. Detailed Implementation

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] To address the issues mentioned in the background description, this application proposes a multimodal face data feature fusion method. Figure 1 This is a flowchart of a multimodal face data feature fusion method according to an embodiment of this application. Figure 2 This is a schematic diagram illustrating the data flow of the multimodal face data feature fusion method according to an embodiment of this application. Figure 1 and Figure 2 As shown, the multimodal face data feature fusion method according to an embodiment of this application includes: S1, acquiring a visible light face image and a near-infrared face image; S2, performing parallel multimodal feature extraction on the visible light face image and the near-infrared face image to obtain the original visible light features and the original near-infrared features of the face; S3, performing cross-modal recalibration on the original visible light features and the original near-infrared features of the face to obtain the recalibrated visible light features and the recalibrated near-infrared features of the face; S4, performing dynamic gating feature fusion on the recalibrated visible light features and the recalibrated near-infrared features of the face to obtain the multimodal fused features of the face.

[0022] In S1, visible light and near-infrared face images are acquired. It's understandable that when face recognition technology faces challenges in complex environments, a single visible light image often suffers from limited information due to variations in illumination, leading to poor recognition performance. To build a robust recognition framework under changing conditions, it's necessary to introduce another data modality that complements visible light information. Near-infrared images, due to their insensitivity to illumination changes, become an ideal choice. Therefore, acquiring visible light and near-infrared face images provides the necessary raw data input, rich in diverse information, for subsequent feature extraction and fusion, thus laying the foundation for the accuracy and robustness of face recognition.

[0023] In one feasible embodiment of this application, the specific process of S1 is as follows: It is worth mentioning that a visible light face image refers to a color face image acquired by a standard image sensor in the visible spectrum range (typically 400 to 760 nanometers). It is rich in high-frequency detail information such as face color and skin texture, but the image quality will be severely affected when the lighting conditions are poor or change drastically. A near-infrared face image, on the other hand, is a grayscale image acquired by a specific image sensor that is sensitive to the near-infrared spectrum (typically 780 to 1100 nanometers). It mainly reflects the reflectivity characteristics of facial tissue, can penetrate the skin surface, has strong robustness to changes in lighting, and is not easily affected by factors such as facial makeup.

[0024] Acquiring these two types of images is accomplished through an integrated binocular imaging module. This module features two independent image sensors arranged side-by-side or coaxially: a standard color image sensor for capturing visible light images, and a monochrome image sensor equipped with a near-infrared filter, specifically designed for capturing near-infrared images. Simultaneously, an active near-infrared light source, such as a near-infrared LED array, is integrated into the module to actively supplement illumination in low-light conditions, ensuring the quality of the near-infrared image acquisition. The fields of view of both sensors are rigorously calibrated to ensure that their imaging range and angle are essentially consistent, thereby acquiring well-aligned pairs of face images.

[0025] When the acquisition process begins, upon detecting a face within the field of view, a synchronous trigger signal is simultaneously sent to both image sensors. This ensures that exposure and acquisition are completed within an extremely short time window, with an error of less than one millisecond, guaranteeing that the captured face is the same at the same moment, with the same pose and expression. The acquired raw data then undergoes preliminary image signal processing. For example, when a user is being identified by an access control device, the device's built-in binocular imaging module is triggered. The module's color sensor captures and outputs a color face image with a resolution of 1920 x 1080 pixels. This image data exists in the form of a three-dimensional matrix, representing the pixel values ​​of the red, green, and blue color channels. Simultaneously, the module's monochrome sensor, assisted by a near-infrared light source, captures and outputs a grayscale face image of the same resolution. Its data exists in the form of a two-dimensional matrix, representing the near-infrared intensity value of each pixel. These two aligned and preprocessed image data constitute the visible light face image and the near-infrared face image.

[0026] In S2, parallel multimodal feature extraction is performed on visible light and near-infrared face images to obtain the original visible light and near-infrared face features. Correspondingly, after obtaining the visible light and near-infrared face images, the original pixel-level data cannot be directly used for effective fusion and recognition because they contain a large amount of redundant information and are located in different data domains. To extract the core information most valuable for face identification and convert it into a unified format suitable for mathematical operations, deep feature extraction is required for both images. Therefore, this application employs parallel multimodal feature extraction to map high-dimensional, heterogeneous image data into a compact feature space rich in semantic information, generating original visible light and near-infrared face features respectively, providing a high-quality, homogeneous data foundation for subsequent cross-modal recalibration and dynamic fusion.

[0027] In one feasible embodiment of this application, step S2, performing parallel multimodal feature extraction on visible light and near-infrared face images to obtain original visible light and near-infrared face features, includes: S21, inputting the visible light face image into a pre-trained backbone network to obtain original visible light face features; and S22, inputting the near-infrared face image into a lightweight backbone network to obtain original near-infrared face features. Specifically, in one feasible embodiment of this application, the pre-trained backbone network is EfficientNet-B0, and the lightweight backbone network is MobileNetV3.

[0028] In the above feasible embodiment, the specific process of S2 is as follows: The specific implementation of this parallel feature extraction process includes two independent but simultaneous branches. The first branch, namely S21, is responsible for processing visible light face images. The core of this branch is a pre-trained backbone network. The pre-trained backbone network refers to a deep convolutional neural network model whose network structure parameters, namely the weights and biases of each layer, have been fully optimized through training on a very large-scale general image dataset (e.g., the ImageNet dataset containing millions of images and thousands of categories). This pre-training enables the network to have powerful and universal visual feature extraction capabilities, and can recognize complex patterns from low-level edges and textures to high-level object parts. In this scheme, EfficientNet-B0 is selected as the pre-trained backbone network. The reason for choosing EfficientNet-B0 is that it achieves an excellent balance in the three dimensions of network depth, width, and input resolution through a compound scaling method, and can achieve extremely high classification accuracy with relatively small computational overhead, making it suitable for extracting fine features from information-dense visible light face images. The EfficientNet-B0 architecture consists of a series of moved-inverted bottleneck convolutional blocks (MBConv). Each MBConv block integrates depthwise separable convolutions to reduce computation, a channel attention mechanism to adaptively adjust the importance of different feature channels, and residual connections to ensure effective gradient propagation. Specifically, the processing flow of an MBConv block includes a 1×1 convolution to expand the number of channels; a 3×3 or 5×5 depthwise separable convolution to extract spatial features on the expanded feature map; a channel attention module that calculates the importance weight of each channel and applies it to the feature map through global average pooling, two fully connected layers, and an activation function; and finally, a 1×1 convolution to reduce the dimensionality of the feature map, projecting the number of channels back to the original dimension and adding it to the input through residual connections. In implementation, the visible light face image obtained in the previous step, such as a 1920 x 1080 pixel color image, is first preprocessed. Preprocessing includes two key operations: resizing and normalization. The image dimensions are uniformly adjusted to the standard input size required by EfficientNet-B0, i.e., 224 by 224 pixels. Then, the pixel values ​​of the image are normalized, scaling them from an integer range of 0 to 255 to a floating-point range of 0 to 1, and the mean of the ImageNet dataset is subtracted and divided by its standard deviation to ensure that the distribution of the input data matches the data distribution during network pre-training. After preprocessing, this 224 by 224 by 3 tensor is input into the EfficientNet-B0 network loaded with pre-trained weights.These pre-trained weights are frozen during the early stages of training the multimodal model, not participating in gradient updates to preserve their powerful generalization feature extraction capabilities. Later in training, these weights can be fine-tuned with a very small learning rate to better adapt to the specific task of face recognition. Data flows through multiple MBConv blocks in the network, with the spatial dimension gradually decreasing and the channel dimension gradually increasing, thus raising the level of feature abstraction. For example, an input tensor of (224,224,3) might become a feature map of (28,28,112) after passing through intermediate layers, ultimately forming a high-dimensional feature map of (7,7,1280) before the global average pooling layer. To obtain the final feature vector, the fully connected classification layer at the end of the network is removed, and the output of the previous layer, the global average pooling layer, is taken. This layer averages the feature map output from the last convolutional layer in spatial dimensions, generating a single value for each channel. For EfficientNet-B0, this produces a 1280-dimensional vector. This 1280-dimensional vector is the original visible light feature of the face, which encapsulates all the key identity information in the original visible light face image.

[0029] The second branch, S22, processes near-infrared face images in parallel. This branch employs a lightweight backbone network. Lightweight backbone networks are deep neural networks specifically designed for efficient operation on computationally limited devices, characterized by a small number of parameters and low computational complexity. The choice of a lightweight network for near-infrared images is based on the practical challenges analyzed in the background: on the one hand, publicly available large-scale, labeled near-infrared face datasets are far less abundant than visible light datasets. Training from scratch using a network with a large number of parameters easily leads to overfitting and poor model generalization ability. On the other hand, near-infrared images themselves have lower information dimensions (such as lack of color) compared to visible light images, and a simpler network structure is sufficient to effectively capture their main discriminative features. This solution selects MobileNetV3 as the lightweight backbone network. MobileNetV3 is a model of lightweight network design. Building upon its predecessor, MobileNet, it introduces a hardware-aware network architecture search technique, combines a channel attention mechanism module, and a more efficient h-swish activation function, achieving higher accuracy while maintaining extremely low latency. Its core structure is also based on inverted residual blocks, significantly reducing parameters and computational cost through depthwise separable convolutions. Its internal bottleneck block structure is similar to EfficientNet's MBConv block, but for extreme efficiency, it uses the h-swish activation function, a computationally cheaper non-linear function that provides performance gains similar to the swish function. In implementation, similar to the visible light branch, the near-infrared face image obtained in the previous step, such as a 1920x1080 pixel grayscale image, is first preprocessed. Since MobileNetV3 also accepts three-channel images as input, the single-channel grayscale image needs to be copied three times to convert it into a pseudo-three-channel image. Next, its size is adjusted to the network's required 224x224 pixels and normalized according to the statistical characteristics of the training set used. The processed 224x224x3 tensor is then fed into the MobileNetV3 network. The network's weights and biases are either trained from scratch on a specially collected near-infrared face dataset or pre-trained on a smaller, related task and then fine-tuned. Data is propagated through a series of bottleneck layers in MobileNetV3, which efficiently extract spatial and channel features. Similar to the EfficientNet-B0 branch, the classifier at the end of the network is removed, and the feature output before the global average pooling layer is extracted.Depending on the specific MobileNetV3 version selected (such as Large or Small), the dimension of the output feature vector will differ. For example, for MobileNetV3-Large, it can be configured to output a 1280-dimensional vector to ensure that its dimension is completely consistent with the feature vector generated by the visible light branch. This 1280-dimensional vector is the original near-infrared feature of the face extracted from the near-infrared image.

[0030] In S3, cross-modal recalibration is performed on the original visible light and near-infrared features of the face to obtain recalibrated visible light and near-infrared features. It is understandable that after extracting the original visible light and near-infrared features in parallel, although two high-level semantic feature vectors are obtained, they are essentially generated in two independent, non-interacting deep networks. As described in the background section, due to differences in network structure, training data, and the amount of information inherent in the modality itself, these two feature vectors exhibit significant imbalances in discriminative power and numerical distribution. Direct fusion would result in strong modality information overwhelming weak modality information. Therefore, this application introduces an intermediate step to address this intermodal asymmetry before final fusion. Through the cross-modal recalibration step, a bridge for information exchange can be established, allowing the features of the two modalities to "examine" and "calibrate" each other. Information from one modality is used to dynamically adjust the representation of the features of the other modality, thereby proactively balancing and enhancing the complementarity of features before fusion, laying the foundation for more effective feature fusion later.

[0031] In one feasible embodiment of this application, Figure 3 This is a flowchart of step S3 in the multimodal face data feature fusion method according to an embodiment of this application. Figure 3 As shown, S3, cross-modal recalibration of the original visible light features and original near-infrared features of the face to obtain recalibrated visible light features and recalibrated near-infrared features of the face, includes: S31, linear projection of the original visible light features of the face to obtain a visible light feature query vector; S32, linear projection of the original near-infrared features of the face to obtain a near-infrared feature key vector and a near-infrared feature value vector; S33, cross-modal calibration of the visible light feature query vector, the near-infrared feature key vector, and the near-infrared feature value vector to obtain an update vector for the RGB modality; S34, aggregation and updating of the update vector for the RGB modality and the original visible light features of the face to obtain the recalibrated visible light features of the face.

[0032] In the above feasible embodiment, the specific process of S3 is as follows: Taking the recalibration of the original visible light features of the face as an example, it includes the following four closely connected sub-steps. First, in S31, the original visible light features of the face are linearly projected to obtain the visible light feature query vector of the face. The linear projection here is implemented through a fully connected layer without an activation function. The input original visible light features of the face are a 1280-dimensional vector. This fully connected layer contains a weight matrix. These parameters are learned through backpropagation during the end-to-end training of the multimodal model. The output dimension of this layer can be pre-set, for example, to 1280 dimensions, to ensure scale consistency during subsequent aggregation. Thus, the original 1280-dimensional visible light features, after this linear transformation, are projected into a 1280-dimensional face visible light feature query vector. Conceptually, this query vector can be understood as an information request issued by the visible light modality, encoding what complementary information the visible light features seek from the near-infrared modality.

[0033] Next, in S32, a similar linear projection is performed on the raw near-infrared facial features to obtain the near-infrared feature key vector and the near-infrared feature value vector. The key difference between this step and the previous one is the use of two independent linear projection layers. The input raw near-infrared facial features (also 1280-dimensional) are simultaneously fed into these two fully connected layers. The first fully connected layer projects it into a 1280-dimensional near-infrared facial feature key vector, which represents the information index that the near-infrared modality can provide. The second fully connected layer projects it into a 1280-dimensional near-infrared facial feature value vector, which represents the specific information content that the near-infrared modality can provide based on the index. The weights of these two linear projection layers... and They are all learnable and independent of each other, allowing the network to flexibly learn how to separate the "keys" for matching and the "values" for passing from the same original features.

[0034] Then, in S33, based on the query, key, and value vectors generated in the first two steps, a core cross-modal calibration calculation is performed to obtain an update vector for the visible light modality. This calculation process employs the mathematical form of a scaled dot product attention mechanism. In one feasible embodiment of this application, S33, cross-modal calibration is performed on the face visible light feature query vector, face near-infrared feature key vector, and face near-infrared feature value vector to obtain an update vector for the RGB modality, including: performing cross-modal calibration on the face visible light feature query vector, face near-infrared feature key vector, and face near-infrared feature value vector using the following formula: ;in, This is a query vector for visible light features of a face. The near-infrared feature key vector of the face. The near-infrared feature vector of the face. for Dimensions This is an update vector for the RGB modality. Specifically, the first step in the calculation is to evaluate the correlation between the query vector and the key vector. This is achieved by calculating the dot product of the transpose of the visible light feature query vector and the near-infrared feature key vector of the face, i.e. The result of this operation is a scalar, and its value directly reflects the degree of matching between the information queried by the visible light features and the information indexed by the near-infrared features. For example, if a visible light image has blurred features in certain areas due to local shadows, its query vector might encode the need for structural information in that area. Meanwhile, the near-infrared feature key vector, unaffected by illumination, might provide clear contour information, resulting in a larger dot product. The second step is to scale the relevance score. The formula... This represents the dimension of the key vector, which is 1280 in this example. Calculate... ,Right now The scaling operation is then performed, dividing the dot product result obtained in the previous step by this value. This scaling operation aims to prevent the dot product result from becoming too large due to high dimensionality, thereby avoiding excessively small gradients in subsequent steps and ensuring the stability of the entire model training process. The third step is to normalize the scaled score using a softmax function. This function transforms the scalar score into an attention weight between 0 and 1. This weight value can be intuitively understood as a quantified representation of the "importance" or "contribution" of the near-infrared mode to the current visible light mode. Finally, the calculated attention weight is compared with the face near-infrared feature vector. Multiplication is performed, which involves extracting the actual information contained in the near-infrared eigenvalue vector proportionally based on the previously calculated importance. If the weight value is high, most of the information in the value vector is retained; conversely, it is suppressed. The final result of this multiplication operation is a 1280-dimensional vector, which is the update vector for the visible light mode. This vector encapsulates the essential information dynamically selected from the near-infrared modes based on requirements. As a targeted supplement to the original visible light features, it will be used for the next step of aggregation and updating.

[0035] Finally, in S34, the update vector obtained in the previous step is aggregated and updated with the original visible light features to obtain the recalibrated visible light features of the face. The aggregation method uses the idea of ​​residual connection, that is, element-wise addition. To achieve addition, it is necessary to ensure that the dimension of the update vector is consistent with the dimension of the original features. Therefore, in the actual design, the linear projection layers in S31 and S32 project the original 1280-dimensional features to 1280 dimensions. Under this setting, the update vector generated in S33 is also 1280-dimensional. Thus, this 1280-dimensional update vector is added element-wise with the 1280-dimensional original visible light features of the face to obtain the final recalibrated visible light features of the face. This additive aggregation not only retains all the information of the original visible light features, but also incorporates incremental information from the near-infrared modality that has been filtered by the attention mechanism, achieving effective enhancement and calibration of the original features.

[0036] It is important to emphasize that the four steps detailed above fully describe how to recalibrate visible light features using near-infrared features. To achieve bidirectional and fair information exchange, a completely symmetrical process is performed in parallel: recalibrating near-infrared features using visible light features. In this symmetrical process, the original near-infrared features of the face are projected into a query vector, while the original visible light features of the face are projected into key and value vectors. After the same attention calculation and aggregation update steps, the recalibrated near-infrared features of the face are finally obtained. These two feature vectors, after bidirectional cross-modal recalibration, already contain complementary information compared to the original features, effectively mitigating the intermodal imbalance.

[0037] In S4, dynamic gating feature fusion is performed on the recalibrated visible light and near-infrared facial features to obtain multimodal facial fusion features. That is, after cross-modal recalibration, the visible light and near-infrared features have achieved a preliminary balance and mutual enhancement in information representation. However, the contribution of these two features to the final face recognition task is not constant but dynamically changes with the quality of the specific input image and scene conditions (such as illumination intensity and facial pose). A fixed fusion strategy that treats the two modalities equally cannot achieve context-adaptive behavior. Therefore, a more intelligent fusion mechanism is needed that can adaptively determine and allocate fusion weights for the two modalities based on the comprehensive information of the current input features. To this end, dynamic gating feature fusion is performed to achieve this data-driven adaptive weighting, ensuring that the advantages of the two modalities are optimally combined in any scenario to generate a final fusion feature with maximized information content and the highest discriminative power.

[0038] In one feasible embodiment of this application, Figure 4 This is a flowchart of step S4 in the multimodal face data feature fusion method according to an embodiment of this application. Figure 4 As shown, S4, dynamic gating feature fusion is performed on the recalibrated visible light features and recalibrated near-infrared features of the face to obtain multimodal fusion features of the face, including: S41, feature splicing is performed on the recalibrated visible light features and recalibrated near-infrared features of the face to obtain multimodal face aggregation features; S42, the multimodal face aggregation features are input into a small gating network to obtain a first dynamic gating weight and a second dynamic gating weight; S43, based on the first dynamic gating weight and the second dynamic gating weight, the weighted sum of the recalibrated visible light features and recalibrated near-infrared features of the face is calculated to obtain the multimodal fusion features of the face.

[0039] In the above feasible embodiment, the specific process of S4 is as follows: First, in S41, the recalibrated visible light features and recalibrated near-infrared features of the face output from the previous step are concatenated to obtain a multimodal face aggregation feature containing all information of both modalities. Continuing from the example of the previous step, the two input feature vectors are both 1280 in dimension. The concatenation operation is performed on the feature dimension, that is, the two 1280-dimensional vectors are concatenated end to end to form a single aggregation feature vector with a dimension of 2560 (1280 plus 1280). This multimodal face aggregation feature serves as the basis for subsequent weight decisions by the gating network because it completely retains all the information of the two modalities after mutual calibration, providing a comprehensive perspective for the network to make comprehensive judgments.

[0040] Next, in S42, the 2560-dimensional multimodal face aggregation feature is input into a small gating network to calculate and generate a first dynamic gating weight and a second dynamic gating weight. This small gating network is essentially a shallow multilayer perceptron, designed to efficiently learn the nonlinear mapping relationship from the aggregation feature to the optimal fusion weight with minimal computational overhead. In one feasible embodiment of this application, S42, inputting the multimodal face aggregation feature into the small gating network to obtain the first dynamic gating weight and the second dynamic gating weight, includes: the small gating network processes the multimodal face aggregation feature using the following formula to obtain the first dynamic gating weight and the second dynamic gating weight, the formula being: ;in, For multimodal face aggregation features, and Here are the weight parameter matrices for the first and second fully connected layers. and These are the bias vectors for the first and second fully connected layers. for Activation function for Activation function It is a dynamic gating weight vector composed of the first and second dynamic gating weights. This small gating network consists of two fully connected layers and two activation functions. The first layer network structure is as follows: .in, It is the input 2560-dimensional multimodal face aggregation features. This is the weight parameter matrix of the first fully connected layer, whose dimension needs to map the 2560-dimensional input to a lower intermediate dimension, such as 128 dimensions. The dimension is 128 by 2560; This corresponds to a 128-dimensional bias vector. These parameters are learned through backpropagation during the entire end-to-end training process of the model. The input multimodal face aggregation features and Perform matrix multiplication and add Then, a nonlinear transformation is performed using a Rectified Linear Unit (ReLU) activation function, outputting a 128-dimensional intermediate feature. The second layer network structure is as follows: ,in This represents the output of the first layer. This is the weight matrix of the second fully connected layer, which maps the 128-dimensional intermediate features to the final weight dimension. Since a weight needs to be generated separately for the visible and near-infrared modes, the output dimension is 2. The dimension is 2 times 128; This corresponds to a 2D bias vector. After the second linear transformation, a vector with two elements is obtained. Finally, a softmax function is applied to this 2D vector. The softmax function can transform a vector of arbitrary real values ​​into a probability distribution, where the two output elements are both between 0 and 1, and their sum is always equal to 1. These two output values ​​are the desired first dynamic gating weights (e.g., assigned to visible light features) and second dynamic gating weights (assigned to near-infrared features). For example, for a set of inputs, the gating network might output a weight vector. The value is [0.7, 0.3], which means that in the current scene, the contribution of visible light features is determined to be 70%, while the contribution of near-infrared features is 30%.

[0041] In particular, after the multimodal features have been recalibrated and the information imbalance problem has been initially resolved, the final fusion of these features becomes crucial to the success of the entire solution. A simple, fixed fusion strategy or a purely data-driven "black box" gating network often struggles to make optimal decisions when faced with complex anomalies such as photo or mask spoofing attacks or sensor failures, because they lack explicit awareness of the intrinsic quality of the current input data and the consistency between modalities. Therefore, a more refined and intelligent fusion weight generation mechanism is desired. To this end, this application establishes a transparent and robust decision-making process through a consensus-aware adaptive gating fusion mechanism for multimodal face aggregation features. The fusion weights generated by this process not only rely on the general patterns learned by the network, but more importantly, they can quantitatively evaluate the intrinsic information quality of each modality and the cross-modal consensus with other modalities in real time. Based on these two key indicators, the final weight allocation is jointly determined, thereby achieving optimal and secure feature fusion in various complex environments.

[0042] Based on this, in a preferred embodiment of this application, S42, the multimodal face aggregation features are input into a small gating network to obtain a first dynamic gating weight and a second dynamic gating weight, including: based on the multimodal face aggregation features, calculating the intrinsic information ratios of the recalibrated visible light features and recalibrated near-infrared features of the face to obtain the intrinsic information ratio of the visible light features and the intrinsic information ratio of the near-infrared features of the face. First, the proportion of information contained in each independent modality to the total amount of aggregated information is quantified as an indicator of its intrinsic information quality. This process is calculated using the following formula: ; ;in, The visible light features of the face after recalibration. For the recalibrated near-infrared features of the face, The spliced ​​multimodal face aggregation features, It is a logarithmic function with base e. To calculate the L2 norm, The ratio of visible light intrinsic information in a human face. The intrinsic information ratio of a face in near-infrared spectroscopy is calculated. The formula first takes the logarithm of the feature vector to amplify the differences between features, then calculates its L2 norm, which can be considered as the information energy carried by the feature vector. The intrinsic information ratio of a modality is obtained by calculating the ratio of the information energy of a single modality to the total information energy of the aggregated features. and These two scalar values ​​directly reflect the cross-validation probability ratio of the original information claims of each modality relative to the internal and external information sets. In other words, the internal information ratio. and It contains the cross-validation probability ratio of the original information claims of each modality relative to the internal-external information set, which can be used as an intrinsic information quality measure of that modality relative to cross-modal fusion. Furthermore, it can be applied to the cross-modal feature distribution by mapping to a compact confidence space in probabilistic form to determine the degree of consensus.

[0043] Based on the ratio of multimodal face aggregation features to the intrinsic information of visible light, cross-modal consensus is calculated on the recalibrated visible light features to obtain the visible light cross-modal consensus. Similarly, based on the ratio of multimodal face aggregation features to the intrinsic information of near-infrared light, cross-modal consensus is calculated on the recalibrated near-infrared features to obtain the near-infrared cross-modal consensus. Next, the consistency between the information of a single modality and the overall information of all modalities is measured. A higher consensus between a modality and the whole indicates greater consistency between the evidence it provides and other modalities, and thus higher reliability. This process is calculated using the following formula: ; ; Here, among them , and This is the weight matrix. , and These are weight vectors, all parameters learned during model training, used to learn the explicit modeling consistency between each modality and the fused modality. Their role is to... , and These features from different sources are projected into the same high-dimensional space through linear transformation to achieve consistency in display modeling. It is the consensus degree of visible light across modes. This is the near-infrared cross-modal consensus degree. The numerator of the formula calculates the norm of the projected individual modal features, while the denominator calculates the norm of the projected aggregate features. The ratio of the two directly reflects their alignment in the semantic space. When the input visible light image is clear and the near-infrared image also confirms the stereo temperature of the face, , and When semantically highly aligned, this ratio approaches 1. Conversely, when the input is a high-resolution photo attack, visible light features... It has very strong facial features, but near-infrared features It will provide a strong non-stereo signal, leading to and As consistency decreases, this ratio will decrease. Finally, this ratio is calculated using the ratio of their respective intrinsic information. or Weighting the index means that modalities with higher intrinsic information quality will have a more significant impact on the final result due to their consensus (or non-consensus). The output... and This refers to the quantized cross-modal consensus of visible light and the cross-modal consensus of near-infrared light.

[0044] The weighted sum of the visible light intrinsic information ratio and visible light cross-modal consensus of the face is calculated, and the weighted sum of the near-infrared intrinsic information ratio and near-infrared cross-modal consensus of the face is also calculated to obtain the visible light weight value and the near-infrared weight value. Then, the two core indicators obtained from the above calculations—information quality and information consistency—are integrated into a comprehensive score. In specific implementation, a set of learnable scalar weight parameters is introduced for each modality; for example, parameters are introduced for the visible light modality. and Introducing parameters for near-infrared modes and The visible light weight value is calculated as follows: Similarly, the near-infrared weight value is calculated as follows: These weight parameters , , and These parameters are learned along with other network parameters through backpropagation during the end-to-end training of the entire model. The values ​​they ultimately learn reflect the optimal strategy found by the model on the global data—that is, to what extent should the intrinsic information of the modality be trusted, and to what extent should it rely on its consensus with other modalities. By... and Perform a weighted summation, and... and By performing a weighted summation, we can ensure that the final weight decision is the result of comprehensively considering the information richness of the modality itself and its synergy with other modalities.

[0045] The visible light and near-infrared weight values ​​are normalized to obtain the first and second dynamic gating weights. That is, since the weight values ​​obtained in the previous step are unconstrained raw scores, they need to be normalized using a normalization function (e.g., the flexible maximum value). The function transforms the data into a probability distribution that sums to 1. After normalization, the two output values ​​are the dynamic weights finally assigned to the recalibrated visible light and near-infrared facial features, which can be directly used for subsequent weighted fusion. Thus, for spoofing attacks (such as photos or masks), this mechanism can identify contradictory signals between modalities and automatically reduce the correlation between the spoofed modality (usually RGB) and the unspoofed modality (usually near-infrared). In other words, it reduces the weight of the spoofed modality while increasing the weight of the modality capable of detecting the attack (such as near-infrared). Simultaneously, when sensors partially fail or are affected by specific environmental noise, the consistency between modalities decreases, leading to reduced consensus and dynamically lowered weights. This intelligently ignores damaged or unreliable data sources, greatly improving the robustness and security of the fusion model.

[0046] Finally, in S43, based on the newly generated dynamic gating weights, the recalibrated visible light features and recalibrated near-infrared features of the face are weighted and summed to calculate the final multimodal fusion features of the face. The calculation process is very straightforward: the first dynamic gating weights ( The first element (0.7) is multiplied by a scalar multiplication with the 1280-dimensional recalibrated visible light features of the face, that is, each element in the feature vector is multiplied by 0.7. Simultaneously, the second dynamic gating weight ( The second element (0.3) is multiplied by a scalar multiplication with the recalibrated 1280-dimensional near-infrared facial features. Then, the two resulting vectors are element-wise added to generate the final 1280-dimensional multimodal fusion feature. This fusion feature dynamically combines the advantages of both modalities, offering richer and more robust information representation compared to any single modality or simple fusion, and can be directly used for subsequent face recognition, comparison, or classification tasks.

[0047] In summary, a multimodal face data feature fusion method based on embodiments of this application has been clarified. It proposes a two-stage strategy of recalibration followed by fusion to address the strong modality dominance problem caused by unbalanced feature representation capabilities in multimodal face recognition. This scheme abandons the simple fusion approach of directly fusing the original features, first introducing a cross-modal recalibration mechanism. Before formal fusion, this mechanism allows features from visible light and near-infrared modalities to interact and mutually calibrate, aiming to adjust and balance the intensity and representation of different modal features, fundamentally alleviating information asymmetry between modalities. Based on this, a dynamic gated fusion network adaptively assigns fusion weights to each recalibrated modal feature according to the characteristics of the current input data. This enables the fusion process to intelligently determine and enhance the contribution of modalities that are more advantageous in specific scenarios (such as near-infrared features in low light), effectively avoiding the continuous dominance of a single modality, thereby achieving full utilization and efficient integration of multimodal information.

[0048] Figure 5 This is a block diagram of a multimodal face data feature fusion system according to an embodiment of this application. Figure 5 As shown, the multimodal face data feature fusion system 100 according to an embodiment of this application includes: a multimodal face image acquisition module 110, used to acquire visible light face images and near-infrared face images; a face image multimodal feature extraction module 120, used to perform parallel multimodal feature extraction on the visible light face images and near-infrared face images to obtain original visible light features and original near-infrared features of the face; a feature recalibration module 130, used to perform cross-modal recalibration on the original visible light features and original near-infrared features of the face to obtain recalibrated visible light features and recalibrated near-infrared features of the face; and a multimodal fusion module 140, used to perform dynamic gating feature fusion on the recalibrated visible light features and recalibrated near-infrared features of the face to obtain multimodal fused features of the face.

[0049] As described above, the multimodal face data feature fusion system 100 according to the embodiments of this application can be implemented in various wireless terminals, such as servers with multimodal face data feature fusion algorithms. In one possible implementation, the multimodal face data feature fusion system 100 according to the embodiments of this application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the multimodal face data feature fusion system 100 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the multimodal face data feature fusion system 100 can also be one of many hardware modules of the wireless terminal.

[0050] Alternatively, in another example, the multimodal face data feature fusion system 100 and the wireless terminal can also be separate devices, and the multimodal face data feature fusion system 100 can be connected to the wireless terminal via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0051] Those skilled in the art will understand that the specific operations of each step in the above-described multimodal face data feature fusion system have been referenced above. Figures 1 to 4 The multimodal face data feature fusion method has been described in detail in the previous section, and therefore, its repeated description will be omitted.

Claims

1. A method for fusing multimodal facial data features, characterized in that, include: Acquire visible light and near-infrared face images; perform parallel multimodal feature extraction on the visible light and near-infrared face images to obtain the original visible light and near-infrared face features; perform cross-modal recalibration on the original visible light and near-infrared face features to obtain recalibrated visible light and near-infrared face features; perform dynamic gating feature fusion on the recalibrated visible light and near-infrared face features to obtain multimodal face fusion features.

2. The multimodal face data feature fusion method according to claim 1, characterized in that, Parallel multimodal feature extraction is performed on visible light face images and near-infrared face images to obtain original visible light features and original near-infrared features of the face. This includes: inputting visible light face images into a pre-trained backbone network to obtain original visible light features of the face; and inputting near-infrared face images into a lightweight backbone network to obtain original near-infrared features of the face.

3. The multimodal face data feature fusion method according to claim 2, characterized in that, The pre-trained backbone network is EfficientNet-B0, and the lightweight backbone network is MobileNetV3.

4. The multimodal face data feature fusion method according to claim 1, characterized in that, Cross-modal recalibration of the original visible light features and original near-infrared features of the face to obtain recalibrated visible light features and recalibrated near-infrared features of the face includes: linear projection of the original visible light features of the face to obtain a visible light feature query vector; linear projection of the original near-infrared features of the face to obtain a near-infrared feature key vector and a near-infrared feature value vector; cross-modal calibration of the visible light feature query vector, the near-infrared feature key vector, and the near-infrared feature value vector to obtain an update vector for the RGB modality; and aggregation and updating of the update vector for the RGB modality and the original visible light features of the face to obtain the recalibrated visible light features of the face.

5. The multimodal face data feature fusion method according to claim 4, characterized in that, Cross-modal calibration is performed on the visible light feature query vector, near-infrared feature key vector, and near-infrared feature value vector of the face to obtain an update vector for the RGB modality. This includes: performing cross-modal calibration on the visible light feature query vector, near-infrared feature key vector, and near-infrared feature value vector of the face using the following formula: ;in, This is a query vector for visible light features of a face. Near-infrared feature key vectors for human faces The near-infrared feature vector of the face. for Dimensions It is an update vector for the RGB modality.

6. The multimodal face data feature fusion method according to claim 1, characterized in that, Dynamically gated feature fusion of recalibrated visible light features and recalibrated near-infrared features of the face is performed to obtain multimodal fusion features of the face. This includes: concatenating the recalibrated visible light features and recalibrated near-infrared features of the face to obtain multimodal face aggregation features; inputting the multimodal face aggregation features into a small gating network to obtain a first dynamic gating weight and a second dynamic gating weight; and calculating a weighted sum of the recalibrated visible light features and recalibrated near-infrared features of the face based on the first dynamic gating weight and the second dynamic gating weight to obtain the multimodal fusion features of the face.

7. The multimodal face data feature fusion method according to claim 6, characterized in that, The multimodal face aggregation features are input into a small gating network to obtain a first dynamic gating weight and a second dynamic gating weight. This includes processing the multimodal face aggregation features using the following formula to obtain the first dynamic gating weight and the second dynamic gating weight: ;in, For multimodal face aggregation features, and Here are the weight parameter matrices for the first and second fully connected layers. and These are the bias vectors for the first and second fully connected layers. for Activation function for Activation function It is a dynamic gating weight vector composed of the first dynamic gating weight and the second dynamic gating weight.

8. A multimodal facial data feature fusion system, characterized in that, include: A multimodal face image acquisition module is used to acquire visible light face images and near-infrared face images; a face image multimodal feature extraction module is used to perform parallel multimodal feature extraction on visible light face images and near-infrared face images to obtain the original visible light features and the original near-infrared features of the face. The feature recalibration module is used to perform cross-modal recalibration on the original visible light features and the original near-infrared features of the face to obtain the recalibrated visible light features and the recalibrated near-infrared features of the face; the multimodal fusion module is used to perform dynamic gating feature fusion on the recalibrated visible light features and the recalibrated near-infrared features of the face to obtain the multimodal fused features of the face.

Images