A face recognition method, device and electronic equipment

Abstract

The application discloses a face recognition method and device and electronic equipment, and relates to the technical field of communication, to solve the problem of poor face recognition effect. The method comprises the following steps: performing feature extraction on first image data and second image data respectively to obtain first modal features and second modal features; performing fusion processing on the first modal features and the second modal features to obtain first fusion features; performing integration processing on the first modal features and the second modal features respectively to obtain first modal integration features and second modal integration features; performing fusion processing on the first modal integration features, the second modal integration features and the first fusion features to obtain second fusion features; and determining a face recognition result according to the second fusion features. According to the application, two-stage fusion is performed on image data of different modalities of a face at a feature layer, deep-level fusion of multi-modal image data can be realized, and the face recognition effect is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a face recognition method, apparatus and electronic device. Background Technology

[0002] Existing 3D face recognition methods are divided into those based on high-quality 3D scan data and those based on low-quality depth image data. The latter typically employs deep learning and modality fusion, using multimodal image data as network input and performing fusion at the input layer, specifically by concatenating the multimodal image data along the channel dimension. However, because lower-level features often contain significant noise, this can lead to the fusion of noise from both modalities in subsequent single-branch networks, amplifying the interference of noise from each modality.

[0003] To address this, existing technologies have proposed a feature layer fusion method, specifically fusing multimodal features extracted from the input layer through a fully connected layer. While this method may lose some input information, it filters out noise from each modality. However, this simple one-stage fusion approach cannot fully utilize and mine complementary information between modalities, thus affecting the face recognition performance. Summary of the Invention

[0004] This application provides a face recognition method, device, and electronic device to solve the problem that existing face recognition methods cannot fully utilize and mine complementary information between modalities, thereby affecting the face recognition effect.

[0005] In a first aspect, embodiments of this application provide a face recognition method, including:

[0006] Feature extraction is performed on the first image data and the second image data respectively to obtain the first modality feature and the second modality feature, wherein the first image data and the second image data are image data of different modalities of the face;

[0007] The first modal feature and the second modal feature are fused to obtain the first fused feature;

[0008] The features of the first modality at different scales and the features of the second modality at different scales are integrated to obtain the first modality integrated feature and the second modality integrated feature.

[0009] The first modality integration feature, the second modality integration feature, and the first fusion feature are fused together to obtain the second fusion feature;

[0010] The face recognition result is determined based on the second fusion feature.

[0011] Optionally, the step of fusing the first modal feature and the second modal feature to obtain the first fused feature includes:

[0012] The first modal feature and the second modal feature are concatenated to obtain a bimodal feature;

[0013] The features of different channels in the dual-modal features are processed by convolution with N convolution branches respectively to obtain N convolution results. Each of the N convolution branches has the same structure, and N is a positive integer greater than 1.

[0014] The N convolution results are integrated to obtain the first fused feature.

[0015] Optionally, the N convolutional branches include a first convolutional block and N sets of second convolutional blocks, wherein the kernel sizes of the first convolutional block and the second convolutional blocks are different;

[0016] The process involves using N convolutional branches to perform convolutional processing on features from different channels in the dual-modal features, resulting in N convolutional results, including:

[0017] The first convolutional block is used to perform convolution processing on the dual-modal features to output the first convolutional features;

[0018] The first convolutional features are processed by grouping convolution through the N groups of second convolutional blocks respectively, and the N convolutional results are output.

[0019] Optionally, the step of extracting features from the first image data and the second image data respectively to obtain first modality features and second modality features includes:

[0020] A single-modal feature extraction network is used to extract features from the first image data and the second image data respectively to obtain the first modal features and the second modal features;

[0021] The single-modal feature extraction network includes M convolutional blocks, M pooling layers, and a multi-scale feature fusion module. The M pooling layers have different parameter settings and are used to process the features output by the M convolutional blocks into the same size. M is a positive integer greater than 1.

[0022] The first convolutional block among the M convolutional blocks is used to perform convolution processing on the input image data. The first pooling layer among the M pooling layers is used to perform pooling processing on the features output by the M convolutional layers respectively. The i-th convolutional block among the M convolutional blocks is used to perform convolution processing on the pooled features output by the (i-1)-th convolutional block among the M convolutional blocks. The j-th pooling layer among the M pooling layers is used to perform pooling processing on the features output by the (j-1)-th convolutional block among the M convolutional blocks. The multi-scale feature fusion module is used to fuse the target convolutional feature and the features output by the second to the M-th pooling layers among the M pooling layers. The target convolutional feature is the feature output by the first pooling layer on the M-th convolutional block among the M pooling layers, where i and j are both integers greater than 1 and less than or equal to M.

[0023] Optionally, after integrating the features of different scales of the first modality feature and the features of different scales of the second modality feature, and before fusing the integrated features of the first modality, the second modality, and the first fused feature, the method further includes:

[0024] The first modality integrated feature, the second modality integrated feature, and the first fusion feature are converted into feature vectors using the Spatial Attention Vectorization (SAV) module, respectively, to obtain the first modality integrated feature vector, the second modality integrated feature vector, and the first fusion feature vector. The SAV module is used to perform weighted processing on the features of each channel in the input features and flatten the weighted features to obtain the feature vector corresponding to the input features.

[0025] The process of fusing the first modality integration feature, the second modality integration feature, and the first fusion feature to obtain the second fusion feature includes:

[0026] The first modality integrated feature vector, the second modality integrated feature vector, and the first fused feature vector are fused to obtain the second fused feature vector;

[0027] The step of determining the face recognition result based on the second fusion feature includes:

[0028] The face recognition result is determined based on the second fused feature vector.

[0029] Optionally, the SAV module includes K sets of convolutional layers and flattening layers, where K is equal to the number of channels of the input features;

[0030] The K sets of convolutional layers are used to perform convolution processing on the features of each channel in the input features, and the flattening layer is used to flatten the features output by the K sets of convolutional layers to obtain the feature vector corresponding to the input features.

[0031] Optionally, the step of integrating features of different scales of the first modality feature and features of different scales of the second modality feature to obtain the first modality integrated feature and the second modality integrated feature includes:

[0032] The first modality feature and the second modality feature are integrated by a single modality feature integration module to obtain the first modality integrated feature and the second modality integrated feature. The single modality feature integration module includes a third convolutional block.

[0033] Secondly, embodiments of this application also provide a face recognition device, comprising:

[0034] The feature extraction module is used to extract features from the first image data and the second image data respectively to obtain first modality features and second modality features, wherein the first image data and the second image data are image data of different modalities of the face;

[0035] The first feature fusion module is used to fuse the first modal feature and the second modal feature to obtain the first fused feature;

[0036] An integration module is used to integrate features of different scales of the first modality feature and features of different scales of the second modality feature to obtain the first modality integrated feature and the second modality integrated feature.

[0037] The second feature fusion module is used to fuse the first modality integrated feature, the second modality integrated feature, and the first fusion feature to obtain the second fusion feature;

[0038] The determination module is used to determine the face recognition result based on the second fused feature.

[0039] Optionally, the first feature fusion module includes:

[0040] A splicing unit is used to splice the first modal feature and the second modal feature to obtain a dual modal feature;

[0041] N convolutional branches are used to perform convolution processing on the features of different channels in the dual-modal features respectively, to obtain N convolution results. Each of the N convolutional branches has the same structure, and N is a positive integer greater than 1.

[0042] An integration unit is used to integrate the N convolution results to obtain the first fusion feature.

[0043] Optionally, the N convolutional branches include a first convolutional block and N sets of second convolutional blocks, wherein the kernel sizes of the first convolutional block and the second convolutional blocks are different;

[0044] The first convolutional block is used to perform convolution processing on the dual-modal features and output the first convolutional feature;

[0045] The N groups of second convolutional blocks are used to perform grouped convolution processing on the first convolutional features and output the N convolutional results.

[0046] Optionally, the feature extraction module is a single-modal feature extraction network, used to extract features from the first image data and the second image data respectively to obtain the first modal features and the second modal features;

[0047] The single-modal feature extraction network includes M convolutional blocks, M pooling layers, and a multi-scale feature fusion module. The M pooling layers have different parameter settings and are used to process the features output by the M convolutional blocks into the same size. M is a positive integer greater than 1.

[0048] The first convolutional block among the M convolutional blocks is used to perform convolution processing on the input image data. The first pooling layer among the M pooling layers is used to perform pooling processing on the features output by the M convolutional layers respectively. The i-th convolutional block among the M convolutional blocks is used to perform convolution processing on the pooled features output by the (i-1)-th convolutional block among the M convolutional blocks. The j-th pooling layer among the M pooling layers is used to perform pooling processing on the features output by the (j-1)-th convolutional block among the M convolutional blocks. The multi-scale feature fusion module is used to fuse the target convolutional feature and the features output by the second to the M-th pooling layers among the M pooling layers. The target convolutional feature is the feature output by the first pooling layer on the M-th convolutional block among the M pooling layers, where i and j are both integers greater than 1 and less than or equal to M.

[0049] Optionally, the face recognition device further includes:

[0050] The Spatial Attention Vectorization (SAV) module is used to convert the first modality integrated feature, the second modality integrated feature, and the first fusion feature into feature vectors respectively, to obtain the first modality integrated feature vector, the second modality integrated feature vector, and the first fusion feature vector. The SAV module is used to perform weighted processing on the features of each channel in the input features, and to flatten the weighted features to obtain the feature vector corresponding to the input features.

[0051] The second feature fusion module is used to fuse the first modality integrated feature vector, the second modality integrated feature vector, and the first fused feature vector to obtain a second fused feature vector;

[0052] The determining module is used to determine the face recognition result based on the second fused feature vector.

[0053] Optionally, the SAV module includes K sets of convolutional layers and flattening layers, where K is equal to the number of channels of the input features;

[0054] The K sets of convolutional layers are used to perform convolution processing on the features of each channel in the input features, and the flattening layer is used to flatten the features output by the K sets of convolutional layers to obtain the feature vector corresponding to the input features.

[0055] Optionally, the integration module is a single-modal feature integration module, used to integrate features of different scales of the first modality feature and features of different scales of the second modality feature to obtain the first modality integrated feature and the second modality integrated feature, wherein the single-modality feature integration module includes a third convolutional block.

[0056] Thirdly, embodiments of this application also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the face recognition method described above.

[0057] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the face recognition method described above.

[0058] In this embodiment, feature extraction is performed on the first image data and the second image data to obtain first modality features and second modality features, wherein the first image data and the second image data are image data of different modalities of the face, respectively. The first modality features and the second modality features are fused to obtain a first fused feature. Features of different scales of the first modality features and features of different scales of the second modality features are integrated to obtain a first modality integrated feature and a second modality integrated feature. The first modality integrated feature, the second modality integrated feature, and the first fused feature are fused to obtain a second fused feature. The face recognition result is determined based on the second fused feature. In this way, by performing two-stage fusion of image data of different modalities of the face at the feature layer, the complementary information between different modalities can be fully utilized and mined, achieving a deeper level of fusion of multimodal image data, thereby improving the face recognition effect. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram of the structure of an existing RGB-D target recognition network based on a two-stream convolutional neural network;

[0061] Figure 2 This is a flowchart of the face recognition method provided in the embodiments of this application;

[0062] Figure 3 This is a schematic diagram of the structure of the lightweight packet fusion network provided in the embodiments of this application;

[0063] Figure 4 This is a schematic diagram of the structure of the single-modal feature extraction network provided in the embodiments of this application;

[0064] Figure 5 This is a schematic diagram of the lightweight multi-branch convolutional modality fusion module provided in the embodiments of this application;

[0065] Figure 6 This is a schematic diagram of the structure of the grouped convolutional modality fusion module provided in the embodiments of this application;

[0066] Figure 7 These are schematic diagrams of two single-branch convolutional structures provided in the embodiments of this application;

[0067] Figure 8This is a structural diagram of the face recognition device provided in the embodiments of this application;

[0068] Figure 9 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] To make the embodiments of this application clearer, the relevant technical knowledge involved in the embodiments of this application will be introduced as follows:

[0071] 3D face recognition is in contrast to 2D face recognition. 2D face recognition uses 2D images, which are essentially projections of 3D objects onto a 2D plane. Since the human face itself possesses 3D attributes, using 3D face data for face recognition is more advantageous. Currently, the common method for acquiring 3D face data is to directly obtain 3D shape data using 3D imaging equipment. 3D imaging equipment mainly includes high-precision 3D scanners and low-precision RGB-D (RGB-D) cameras. The former can obtain high-quality 3D face data, but the equipment cost is high, the acquisition time is long, and the subject needs to maintain a fixed posture during the acquisition process; the latter has low equipment cost, fast acquisition speed, and is easy to use on a large scale, but the acquired depth map quality is lower and has more noise. Common forms of 3D face data representation include point clouds, depth maps, normal maps, and polygon meshes. Based on the source of the 3D face data, 3D face recognition methods can be divided into face recognition methods based on high-quality 3D scan data and face recognition methods based on low-quality depth image data. The technical solution proposed in this application focuses on face recognition methods based on low-quality depth image data.

[0072] Compared to traditional face recognition methods, deep learning-based face recognition methods have significant advantages in image processing. A key step in traditional methods is finding robust feature points and descriptors based on the geometric information of 3D face data. While these methods can achieve good recognition performance, they involve relatively complex algorithmic operations when detecting features. In contrast, deep learning-based methods can learn robust face representations by training deep neural networks on large datasets. The technical solution proposed in this application mainly uses deep neural networks, particularly convolutional neural networks, to extract features from deep face images and perform face recognition.

[0073] Because depth maps obtained from low-precision RGB-D cameras are of low quality and contain significant noise, directly using such low-quality depth maps for face recognition often results in low accuracy. To achieve higher face recognition accuracy, many studies have shown that fusing multiple modalities (such as depth maps, normal maps, and RGB maps) for face recognition yields significantly better results than using only single-modal data. Fusion methods typically involve three levels: the input layer, the feature layer, or the decision layer. Considering the characteristics of low-quality 3D face data and the properties of different modal fusion methods, the technical solution proposed in this application uses multimodal face data fusion at the feature layer. In some embodiments, depth map modality and normal map modality can be fused at the feature layer.

[0074] Some researchers have provided an efficient and accurate deep learning solution for 3D face recognition based on low-quality data. They have proposed a lightweight and powerful deep neural network model based on convolutional neural networks, called Lightweight and Efficient Deep Approach to Recognizing Low-quality 3D Faces (Led3D).

[0075] The Led3D model first concatenates a single-channel depth map and a three-channel normal map to obtain a four-channel image. This four-channel image is then input into a backbone network consisting of four convolutional blocks for feature extraction. Next, it is processed through a Multi-Scale Feature Fusion (MSFF) module and a Spatial Attention Vectorization (SAV) module. The MSFF module fuses the multi-scale features from each convolutional block, providing a comprehensive representation of the fused features. The SAV module emphasizes important spatial information. Specifically, the MSFF module downsamples the feature maps output from the four convolutional blocks to the same size using a max-pooling layer, then concatenates them along the channel dimension to form multi-scale features. These multi-scale features are then integrated through another convolutional block. The SAV module adds an attention weight map of the same spatial size to each channel's feature map, and then performs a weighted sum of the feature maps along the spatial dimension based on the attention weight map to obtain the final feature vector.

[0076] LED3D uses depth map and normal map modalities as network inputs and fuses them at the input layer by directly concatenating the depth map and normal map along the channel dimension. While this fusion method is simple, direct, and easy to implement, lower-level network features often contain significant noise. This means that fusion at the input stage may amplify the noise from both modalities in subsequent single-branch networks. To address this shortcoming, this application proposes fusing the depth map and normal map modalities at the feature layer. While feature layer fusion may lose some input information, it also filters out noise from each modality, often resulting in better performance.

[0077] Some researchers have also proposed a deep learning method for face recognition that fuses RGB modalities and depth map modalities at the feature level. The network structure used in this method is as follows: Figure 1 As shown, two independent convolutional neural networks are used to extract RGB modal features and depth map modal features respectively. The extracted features of the two modalities are concatenated after passing through two fully connected layers, and then fused through a fully connected layer. Finally, the fused features are fed into the classification layer for face recognition.

[0078] This method uses RGB modalities and depth map modalities for feature fusion at the feature level. However, this modal fusion only occurs in the fully connected layer before the classification layer. This simple one-stage fusion approach cannot fully utilize and mine the complementary information between modalities, and using fully connected layers for modal fusion significantly increases the number of network parameters and computational cost, increasing the risk of overfitting, slowing down the model's inference speed, and hindering the deployment and implementation of the network on edge mobile devices. To address the shortcomings of this method, this application proposes a two-stage feature fusion strategy to fully mine the complementary information between different modalities. In some embodiments, a lightweight multi-branch convolutional structure is used to achieve effective modal feature fusion while reducing the computational overhead of modal feature fusion, facilitating the subsequent deployment and implementation of the model.

[0079] The face recognition method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0080] See Figure 2 , Figure 2 This is a flowchart of the face recognition method provided in the embodiments of this application, such as... Figure 2 As shown, it includes the following steps:

[0081] Step 201: Extract features from the first image data and the second image data respectively to obtain the first modality features and the second modality features, wherein the first image data and the second image data are image data of different modalities of the face.

[0082] In this embodiment of the application, in order to obtain better face recognition accuracy and reduce modal noise interference, multimodal face image features can be fused at the feature layer to perform three-dimensional face recognition. This means obtaining face image data of different modalities, extracting features from the face image data of different modalities respectively, and then fusing the face features of different modalities.

[0083] The first image data and the second image data mentioned above are image data of different modalities of the face, specifically low-quality three-dimensional image data of different modalities of the face. For example, they can be the depth map and normal map of the face, or the depth map and RGB map of the face. The depth map is obtained by acquiring the face through an RGB-D camera and represents the distance of each point in the face to the camera. The normal map is calculated from the depth map and represents the orientation information of each point in the face in three-dimensional space.

[0084] In this step, feature extraction can be performed on the first image data and the second image data respectively to obtain first modality features and second modality features. The first modality features are image features extracted from the first image data, and the second modality features are image features extracted from the second image data. Specifically, feature extraction networks, such as convolutional networks and encoding networks, can be used to perform feature extraction. For example, the depth map and normal map of a face can be extracted separately through a feature extraction network to obtain depth map modality features and normal map modality features.

[0085] The depth map and normal map images can be standardized to 128×128 pixels, with the depth map being a single-channel image and the normal map being a three-channel image. The extracted depth map modal features can be represented as FM. d ∈R C×H×W The modal characteristics of the normal plot are represented as FM n ∈R C×H×W Where C represents the number of channels in the output feature map, and H and W represent the spatial size of the output feature map, which are the height and width, respectively.

[0086] Optionally, step 201 includes:

[0087] A single-modal feature extraction network is used to extract features from the first image data and the second image data respectively to obtain the first modal features and the second modal features;

[0088] The single-modal feature extraction network includes M convolutional blocks, M pooling layers, and a multi-scale feature fusion module. The M pooling layers have different parameter settings and are used to process the features output by the M convolutional blocks into the same size. M is a positive integer greater than 1.

[0089] The first convolutional block among the M convolutional blocks is used to perform convolution processing on the input image data. The first pooling layer among the M pooling layers is used to perform pooling processing on the features output by the M convolutional layers respectively. The i-th convolutional block among the M convolutional blocks is used to perform convolution processing on the pooled features output by the (i-1)-th convolutional block among the M convolutional blocks. The j-th pooling layer among the M pooling layers is used to perform pooling processing on the features output by the (j-1)-th convolutional block among the M convolutional blocks. The multi-scale feature fusion module is used to fuse the target convolutional feature and the features output by the second to the M-th pooling layers among the M pooling layers. The target convolutional feature is the feature output by the first pooling layer on the M-th convolutional block among the M pooling layers, where i and j are both integers greater than 1 and less than or equal to M.

[0090] In one implementation, to ensure good feature extraction performance and accurate extraction of features from each modality, thereby guaranteeing face recognition accuracy, a single-modal feature extraction network composed of multiple convolutional blocks, pooling layers, and a multi-scale feature fusion module can be used to extract features from the image data of each modality. The pooling layers can be max pooling layers, and the multi-scale feature fusion module can be an MSFF module. For example, as... Figure 3 As shown, the depth map and normal map of a face can be extracted separately through a single-modal feature extraction network to obtain depth map modal features and normal map modal features.

[0091] The structure of the single-modal feature extraction network can be as follows: Figure 4As shown, the system consists of four alternating convolutional blocks, max-pooling layers, and a multi-scale feature fusion (MSFF) module. Blocks 1, 2, 3, and 4 are the four convolutional blocks, and MaxPooling1, 2, 3, and 4 are the four max-pooling layers. Each convolutional block contains a 3×3 convolutional layer (Conv), a batch normalization layer (BN), and a rectified linear unit (ReLU) activation function layer. The spatial size of the input and output feature maps remains unchanged for each convolutional block, with the number of output channels being 32, 64, 128, and 256, respectively. The max-pooling layer after each convolutional block is used for spatial downsampling. The MSFF module fuses the multi-scale features from each convolutional block. Specifically, the MSFF module downsamples the feature maps output from the four convolutional blocks to the same size using the max-pooling layer, and then concatenates them along the channel dimension to form multi-scale features.

[0092] The single-modal feature extraction network receives the original depth map I respectively. d ∈R 1×128×128 and normal diagram I n ∈R 3 ×128×128 As input, features of each modality are extracted, and the modal features FM of each modality are output. d ∈R 480×8×8 FM n ∈R 480×8×8 The parameter settings for the entire single-mode extraction network are shown in Table 1 below.

[0093] Table 1. Parameter settings for the single-modal feature extraction network

[0094]

[0095] Where Conv represents a convolutional layer and BN represents a batch normalization layer; the output dimensions from left to right are: number of channels, height, and width.

[0096] Thus, in this embodiment, by utilizing Figure 4 The single-modal feature extraction network shown extracts features for each modality, ensuring that the extracted features have strong representational capabilities.

[0097] Step 202: Perform a fusion process on the first modal feature and the second modal feature to obtain the first fused feature.

[0098] In this embodiment, to achieve deeper modal information fusion, the fusion of different modal features is performed in two stages. After extracting each modal feature, a first-stage fusion can be performed, that is, the first modal feature and the second modal feature can be fused for the first time to obtain the first fused feature. The fusion method in the first stage can be a feature fusion method such as convolution or fully connected layers. Through this stage of fusion, the correlation between different modalities can be captured, and the complementary information between modalities can be mined.

[0099] Optionally, step 202 includes:

[0100] The first modal feature and the second modal feature are concatenated to obtain a bimodal feature;

[0101] The features of different channels in the dual-modal features are processed by convolution with N convolution branches respectively to obtain N convolution results. Each of the N convolution branches has the same structure, and N is a positive integer greater than 1.

[0102] The N convolution results are integrated to obtain the first fused feature.

[0103] In one implementation, to reduce the computational overhead of modal feature fusion, i.e., to reduce the number of parameters and computational cost of the fusion network, this application designs a lightweight multi-branch convolutional modal fusion module to achieve the first-stage fusion of various modal features.

[0104] Specifically, in the first stage of fusion, the first modal feature and the second modal feature are first concatenated along the channel dimension to obtain bimodal features, for example, such as... Figure 3 As shown, the depth map modal features FM d and normal plot modal characteristics FM n By concatenating the features along the channel dimension, we obtain the bimodal feature FM. dn =Concatenate(F d ,F n )∈R 2C×H×W Then, the dual-modal feature FM dn The data is fed into a lightweight multi-branch convolutional mode fusion module for mode fusion, resulting in the first-stage mode fusion feature FM. fuse =f θ (FM dn )∈R 2C×H×W , where f represents the lightweight multi-branch convolutional modality fusion module, and θ represents the learnable parameters of this module. The lightweight multi-branch convolutional modality fusion module can achieve multi-scale fusion of features between modalities with as few parameters and computational cost as possible.

[0105] The structure of the lightweight multi-branch convolutional modality fusion module can be as follows: Figure 5 As shown, it includes N parallel convolutional branches and a consolidation layer, which can be implemented by concatenation and convolutional layers. From Figure 5 As can be seen from the diagram, the modality fusion method in this implementation uses a multi-branch convolutional structure. Compared to a single-branch convolutional structure, this implementation feeds the input bimodal features into multiple parallel convolutional branches. Each convolutional branch shares the same network topology, and the number of branches is controlled by the hyperparameter N, such as 32 or 64. Each convolutional branch can consist of a 1×1 convolutional layer and a 3×3 convolutional layer. The 1×1 convolutional layer is used to integrate features at different scales from the depth map modality and the normal map modality, while the 3×3 convolutional layer is used to further fuse different modal features in spatial and channel dimensions. Each convolutional branch performs convolution processing on features from different channels in the bimodal features. For example, the depth map modality feature FM... d ∈R 480×8×8 and normal plot modal characteristics FM n ∈R 480×8×8 After concatenation along the channel dimension, the bimodal feature FM is obtained. dn ∈R 960×8×8 Dual-modal characteristics FM dn ∈R 960×8×8 The data is fed into the lightweight multi-branch convolutional modality fusion module and processed in parallel by the N convolutional branches. Each convolutional branch processes the dual-modality feature FM. dn ∈R 960×8×8 The features span 960 / N channels. Finally, the outputs of the N convolutional branches are combined along the channel dimension, such as concatenating the output features of the N convolutional branches along the channel dimension, and then integrating the features of different convolutional branches through a final 1×1 convolutional layer to output the final fused feature FM. fuse ∈R 960×8×8 In addition, each convolutional layer in the lightweight multi-branch convolutional modality fusion module is followed by a batch normalization (BN) layer and a ReLU activation function layer.

[0106] In this implementation, since each convolutional branch first uses a 1×1 convolutional layer to reduce the dimensionality of the input channels, this is equivalent to projecting the input features onto several different feature subspaces. In these different feature subspaces, the network can learn and discover more diverse feature representations, providing a more diverse perspective for intermodal feature fusion. Furthermore, this multi-branch operation also serves as a regularization mechanism for the network. Each additional branch allows the network to learn more sparsely related features. Because while there is complementary information between different modal data such as depth maps and normal maps, there is also a large amount of redundant information, this sparsity significantly reduces the risk of overfitting.

[0107] In this implementation, by using a multi-branch convolutional structure to fuse multimodal features, multi-scale fusion of features between modalities can be achieved with as few parameters and computational cost as possible. This can significantly reduce the additional computational overhead caused by modal fusion and improve the inference speed of the model.

[0108] Optionally, the N convolutional branches include a first convolutional block and N sets of second convolutional blocks, wherein the kernel sizes of the first convolutional block and the second convolutional blocks are different;

[0109] The process involves using N convolutional branches to perform convolutional processing on features from different channels in the dual-modal features, resulting in N convolutional results, including:

[0110] The first convolutional block is used to perform convolution processing on the dual-modal features to output the first convolutional features;

[0111] The first convolutional features are processed by grouping convolution through the N groups of second convolutional blocks respectively, and the N convolutional results are output.

[0112] In one implementation, to further reduce the complexity of the multi-convolutional branch structure, in engineering practice, grouped convolution can be used to implement the previous implementation. Figure 5 The multi-branch convolutional structure is shown. An equivalent structure implemented using grouped convolutions can be seen as follows: Figure 6 As shown, the N convolutional branches can be implemented by a first convolutional block and N sets of second convolutional blocks. The kernel sizes of the first convolutional block and the second convolutional blocks are different. The first convolutional block can be composed of a 1×1 convolutional layer, a BN layer and a ReLU activation function layer, while the second convolutional block is composed of a 3×3 convolutional layer, a BN layer and a ReLU activation function layer.

[0113] In this way, the bimodal features can be convolved by the first convolutional block to integrate features of different modalities and scales, and the number of input channels can be reduced in dimensionality. Then, the features output by the first convolutional block are grouped and convolved by the N groups of second convolutional blocks. Each group of second convolutional blocks processes the input features of different channels, further realizing the fusion of different modal features in spatial and channel dimensions. Finally, a 1×1 convolutional layer is used to integrate the features output by different groups of convolutional blocks to output the final fused features.

[0114] In this implementation, grouped convolution can be used to achieve this conveniently. Figure 5 The lightweight multi-branch convolutional modality fusion module shown takes bimodal features (FM) as input. dn ∈R 960×8×8 The output is the feature FM after dual-modal fusion. fuse ∈R 960×8×8 The specific parameter settings for the lightweight multi-branch convolutional modality fusion module in this embodiment are shown in Table 2 below:

[0115] Table 2 Parameter Settings for Lightweight Multi-Branch Convolutional Modality Fusion Module

[0116]

[0117] The three blocks mentioned above correspond to three convolutional blocks in turn, and the hyperparameter N represents the number of branches.

[0118] Thus, by adopting the grouped convolution method in this embodiment to achieve an equivalent multi-branch convolution structure, the complexity of the multi-branch convolution structure can be further reduced, and the number of parameters and computation can be reduced.

[0119] Furthermore, to verify the lightweight advantage of this multi-branch convolutional structure, embodiments of this application compare the number of parameters and computational cost (e.g., floating-point operations per second, FLOPs) of the multi-branch convolutional structure with those of a single-branch convolutional structure with the same input-output dimensions. The comparison diagrams of the two single-branch convolutional structures are shown below. Figure 7 As shown.

[0120] The number of parameters and computational cost (FLOPs) for these three convolutional structures were calculated respectively, and the results are shown in Table 2 below:

[0121] Table 3 Comparison of parameter quantity and computational cost of different convolutional structures

[0122] Model (Module) Params (millions) FLOPs (millions of floating-point operations) Multi-branch convolutional structure (N=32) 0.988 63.25 Single-branch convolutional structure ( Figure 7 (a)) 2.997 191.82 Single-branch convolutional structure ( Figure 7 (b)) 8.295 530.9

[0123] In Table 3, the number of branches selected for the multi-branch convolutional structure is N=32. As can be seen from Table 3, the number of parameters and computational cost of the multi-branch convolutional structure are significantly less than those of the single-branch convolutional structure with the same input and output dimensions. Furthermore, the number of parameters and computational cost decreases further as the number of branches increases, which meets the design requirements of the lightweight multi-branch convolutional modal fusion module in this application.

[0124] Step 203: Integrate the features of different scales of the first modality feature and the features of different scales of the second modality feature to obtain the first modality integrated feature and the second modality integrated feature.

[0125] In this embodiment of the application, after feature extraction is performed on the first image data and the second image data respectively, in order to achieve multimodal feature fusion in the second stage, the features of each single modality at different scales are integrated to generate more distinctive three-dimensional face features. Subsequently, the integrated modal features are fused with the multimodal features fused in the first stage to achieve a deeper level of modal information fusion.

[0126] Therefore, in this step, features of different scales of the first modality feature can be integrated to obtain the first modality integrated feature of the first image data, and features of different scales of the second modality feature can be integrated to obtain the second modality integrated feature of the second image data. The integration method can be implemented using convolution, fully connected layers, or similar methods. It should be noted that step 203 can be executed in parallel with step 202, and the specific execution is not limited.

[0127] For example, for the extracted depth map modal features FM d ∈R C×H×W and normal plot modal characteristics FM n ∈R C×H×W After being integrated into single-modal models, the integrated depth map modal features (FM) are output. d′ ∈R C×H×W and the integrated normal map modal features FM n′ ∈R C×H×W .

[0128] Optionally, step 203 includes:

[0129] The first modality feature and the second modality feature are integrated by a single modality feature integration module to obtain the first modality integrated feature and the second modality integrated feature. The single modality feature integration module includes a third convolutional block.

[0130] In one implementation, a single-modal feature integration module composed of convolutional block structures can be used to integrate various single-modal features. Specifically, the single-modal feature integration module includes a third convolutional block, which can be composed of a 3×3 convolutional layer, a batch normalization layer (BN), and a ReLU activation function layer.

[0131] For example, such as Figure 3 As shown, depth map modal features and normal map modal features can be integrated at different scales through a single modal feature integration module to obtain integrated depth map modal features and normal map modal features.

[0132] The function of the single-modal feature integration module is to integrate the features of each modality obtained from the multi-scale feature fusion (MSFF) module at different scales, generating more discriminative 3D face features. Simultaneously, this module runs in parallel with the first-stage modality fusion module, thus fully preserving the features of each modality. The parameter settings for the single-modal feature integration module are shown in Table 4 below:

[0133] Table 4 Parameter Settings for Single-Modal Feature Integration Module

[0134]

[0135] The single-modal feature integration module accepts Figure 4 The output FM of the multi-scale feature fusion (MSFF) module in the middle d FM n ∈R 480×8×8 As input to the module, the module output is a more discriminative feature FM, which integrates features from different scales of each modality. d′ FM n′ ∈R 480×8×8 .

[0136] In this way, by using the single-modal feature integration module of the convolutional block structure to integrate each single-modal feature, not only is the number of parameters and computational cost small, but more discriminative features can also be generated for each modality.

[0137] Step 204: Perform fusion processing on the first modality integration feature, the second modality integration feature, and the first fusion feature to obtain the second fusion feature.

[0138] In this step, based on the integrated single-modal features and the bimodal features fused in the first stage, a second-stage fusion of these features can be performed to achieve a deeper level of fusion of multimodal features, fully utilize and mine the complementary information between different modalities, and obtain three-dimensional face features with stronger representation capabilities.

[0139] Specifically, the first modality integration feature, the second modality integration feature, and the first fusion feature can be spliced ​​together to achieve a second fusion of features between modalities, resulting in a second fusion feature.

[0140] For example, the integrated depth map modal features FM d′ ∈R 480×8×8 Integrated normal map modal features (FM) n′ ∈R 480×8×8 Feature FM after fusion with the first stage mode fuse ∈R 960×8×8 By splicing, the second fusion feature FM is obtained. fuse′ =Concatenate(FM) d′ FM n′ FM fuse (), as the final representation of facial features.

[0141] This stage of fusion can maintain the distinguishability between different modalities and reduce the loss of single-modal information caused by modality fusion.

[0142] Step 205: Determine the face recognition result based on the second fusion feature.

[0143] In this step, face recognition can be performed based on the final facial feature representation, i.e., the second fused feature, to determine the face recognition result. Specifically, it can be as follows: Figure 3 As shown, the final facial feature representation FM fuse′ The classifiers are classified using a fully connected layer (FC) and a softmax layer to obtain the corresponding face ID.

[0144] Optionally, after step 204 and before step 205, the method further includes:

[0145] The first modality integrated feature, the second modality integrated feature, and the first fusion feature are converted into feature vectors using the Spatial Attention Vectorization (SAV) module, respectively, to obtain the first modality integrated feature vector, the second modality integrated feature vector, and the first fusion feature vector. The SAV module is used to perform weighted processing on the features of each channel in the input features and flatten the weighted features to obtain the feature vector corresponding to the input features.

[0146] Step 204 includes:

[0147] The first modality integrated feature vector, the second modality integrated feature vector, and the first fused feature vector are fused to obtain the second fused feature vector;

[0148] Step 205 includes:

[0149] The face recognition result is determined based on the second fused feature vector.

[0150] In one implementation, before performing the second stage of modal feature fusion, the SAV module can be used to vectorize each feature map to be fused, namely the first modal integration feature, the second modal integration feature, and the first fused feature, so as to emphasize the important spatial information in each feature through the SAV, and ensure that the second stage achieves a better feature fusion effect.

[0151] The SAV module specifically adds an attention weight map of the same spatial size to each channel of the input feature map. Then, it performs a weighted summation of the feature maps of each channel in the spatial dimension based on the attention weight maps, and flattens the spatial dimension to obtain the final feature vector corresponding to the input feature map.

[0152] In this way, the SAV module can be used to convert the first modality integrated feature, the second modality integrated feature, and the first fusion feature into feature vectors respectively, to obtain the first modality integrated feature vector, the second modality integrated feature vector, and the first fusion feature vector; then the first modality integrated feature vector, the second modality integrated feature vector, and the first fusion feature vector are concatenated to obtain the second fusion feature vector; finally, the face recognition result is determined based on the second fusion feature vector.

[0153] For example, such as Figure 3 As shown, FM is achieved by using three SAV modules. d′ FM n′ and FM fuse Convert them into feature vectors FV respectively d′ FV n′ ∈R C FV fuse ∈R 2C Finally, these three feature vectors are concatenated together to obtain the feature vector FV. fuse′ =Concatenate(FV d′ ,FV n′ ,FV fuse This serves as the final facial feature representation, completing the second stage of modal feature fusion. Based on the final facial feature representation vector FV... fuse′ The face ID is determined by classifying the faces using a fully connected layer (FC) and SoftMax.

[0154] In this way, by implementing this method, important spatial information in each feature can be emphasized through SAV, ensuring that the second-stage fusion of the inter-modal features after SAV processing can obtain a more representative final facial feature representation, thereby helping to improve the accuracy of facial recognition.

[0155] Optionally, the SAV module includes K sets of convolutional layers and flattening layers, where K is equal to the number of channels of the input features;

[0156] The K sets of convolutional layers are used to perform convolution processing on the features of each channel in the input features, and the flattening layer is used to flatten the features output by the K sets of convolutional layers to obtain the feature vector corresponding to the input features.

[0157] In one implementation, the SAV module can also be implemented using grouped convolutions to reduce the number of parameters and computational cost. Specifically, the SAV module includes K groups of convolutional layers and flattening layers. The value of K can be determined based on the number of channels of the input features. Taking an input channel of 960 as an example, the specific parameter settings are shown in Table 5 below:

[0158] Table 5 SAV Module Parameter Settings

[0159]

[0160] For example, such as Figure 3 As shown, the integrated depth map modal features FM d′ ∈R 480×8×8 and normal plot modal characteristics FM n′ ∈R 480×8×8 and the characteristic FM after dual-modal fusion fuse ∈R 960×8×8 After passing through three SAV modules, each module outputs its respective feature vector FV. d′ FV n′ ∈R 480 FV fuse ∈R 960 Finally, the three feature vectors FV d′ FV n′ FV fuse By stitching them together, a second fusion between modalities is achieved, resulting in the final facial feature representation (FV). fuse′ ∈R 1920 .

[0161] In this way, this implementation method can help better integrate features between modalities while reducing the complexity of modules, as well as the number of module parameters and computational load.

[0162] It should also be noted that, in the embodiments of this application, when using Figure 3When performing 3D face recognition using the lightweight group fusion network model structure shown, the final face feature representation FV can be obtained during the model training phase. fuse′ Classification is performed using a fully connected layer and SoftMax, with the classification loss L2 calculated using the cross-entropy loss function. Furthermore, to enhance the discriminative power of each modality's features and preserve as much complete information as possible, the face feature vectors (FV) calculated separately for the depth map modality and the normal map modality are also used. d′ FV n′ The losses L1 and L3 are calculated using a fully connected layer and SoftMax classification, respectively, and the cross-entropy loss function is used. The total loss L during the final training phase is represented as the sum of these three losses, i.e., L = L1 + L2 + L3.

[0163] In addition, to alleviate the overfitting problem, Figure 3 The model shown can also add a random dropout layer before each fully connected layer, with the hyperparameter p = 0.2 for each dropout layer. The parameter settings for the three fully connected layers are shown in Table 6 below:

[0164] Table 6. Parameter Settings for Fully Connected Layers

[0165]

[0166]

[0167] Where C represents the number of face IDs trained.

[0168] The face recognition method of this application embodiment extracts features from first image data and second image data respectively to obtain first modality features and second modality features, wherein the first image data and the second image data are image data of different modalities of the face. The first modality features and the second modality features are fused to obtain a first fused feature. Features of different scales of the first modality features and features of different scales of the second modality features are integrated to obtain a first modality integrated feature and a second modality integrated feature. The first modality integrated feature, the second modality integrated feature, and the first fused feature are fused to obtain a second fused feature. The face recognition result is determined based on the second fused feature. Thus, by performing two-stage fusion of image data of different modalities of the face at the feature layer, complementary information between different modalities can be fully utilized and mined, achieving a deeper level of fusion of multimodal image data, thereby improving the face recognition effect.

[0169] like Figure 3As shown in the figure, this application also provides a face recognition network, including two single-modal feature extraction networks, two single-modal feature integration modules, a multi-branch convolutional modality fusion module, two concatenation modules, three SAV modules, and three fully connected layers. The connection relationships of each module are as follows: Figure 3 As shown, the functions and signal processing flow of each module can be found in the relevant descriptions in the foregoing embodiments. To avoid repetition, they will not be repeated here.

[0170] This application defines a more general 3D face modality fusion and recognition model, proposes a two-stage modality fusion method, and realizes a deeper level of fusion between depth map modality and normal map modality. It also proposes a lightweight multi-branch convolutional structure, which reduces the computational overhead of modality fusion while achieving modality fusion with more diverse angles. This fusion method can not only be used for the fusion of depth map and normal map, but can also be extended to two other different modality fusion tasks.

[0171] This application embodiment chooses to perform modality fusion at the feature layer. The advantage of fusion at the feature layer is that it utilizes a pre-existing single-modal feature extraction network to filter noise from each modality, achieving a deeper level of intermodal fusion. Simultaneously, to more fully and effectively utilize information between different modalities, this application embodiment proposes a two-stage modality feature fusion scheme. The first stage of fusion primarily captures the correlation between different modalities and mines complementary information; the second stage of fusion primarily maintains the distinguishability between different modalities and reduces the loss of single-modal information caused by modality fusion. In summary, this application embodiment can more effectively utilize multimodal information and achieve higher-precision face recognition on low-quality 3D face data.

[0172] This application employs an innovative multi-branch convolutional structure to achieve the first-stage modal feature fusion. Compared to using fully connected layers, the multi-branch convolutional structure significantly reduces the additional computational overhead associated with modal fusion, thereby improving the model's inference speed. Furthermore, compared to single-branch convolutional structures, the multi-branch convolutional structure encourages the model to explore inter-modal fusion from more diverse perspectives. Therefore, this application's embodiment achieves deeper and more diverse modal feature fusion with minimal parameters and computational cost, facilitating high-precision 3D face recognition and subsequent model deployment and implementation.

[0173] This application also provides a face recognition device. See [link to relevant documentation]. Figure 8 , Figure 8This is a structural diagram of the face recognition device provided in the embodiments of this application. Since the principle of the face recognition device in solving the problem is similar to that of the face recognition method in the embodiments of this application, the implementation of this face recognition device can refer to the implementation of the method, and the repeated parts will not be described again.

[0174] like Figure 8 As shown, the face recognition device 800 includes:

[0175] The feature extraction module 801 is used to extract features from the first image data and the second image data respectively to obtain first modality features and second modality features, wherein the first image data and the second image data are image data of different modalities of the face;

[0176] The first feature fusion module 802 is used to fuse the first modal feature and the second modal feature to obtain the first fused feature;

[0177] The integration module 803 is used to integrate features of different scales of the first modal feature and features of different scales of the second modal feature to obtain the first modal integrated feature and the second modal integrated feature.

[0178] The second feature fusion module 804 is used to fuse the first modality integrated feature, the second modality integrated feature and the first fusion feature to obtain the second fusion feature;

[0179] The determination module 805 is used to determine the face recognition result based on the second fusion feature.

[0180] Optionally, the first feature fusion module 802 includes:

[0181] A splicing unit is used to splice the first modal feature and the second modal feature to obtain a dual modal feature;

[0182] N convolutional branches are used to perform convolution processing on the features of different channels in the dual-modal features respectively, to obtain N convolution results. Each of the N convolutional branches has the same structure, and N is a positive integer greater than 1.

[0183] An integration unit is used to integrate the N convolution results to obtain the first fusion feature.

[0184] Optionally, the N convolutional branches include a first convolutional block and N sets of second convolutional blocks, wherein the kernel sizes of the first convolutional block and the second convolutional blocks are different;

[0185] The first convolutional block is used to perform convolution processing on the dual-modal features and output the first convolutional feature;

[0186] The N groups of second convolutional blocks are used to perform grouped convolution processing on the first convolutional features and output the N convolutional results.

[0187] Optionally, the feature extraction module 801 is a single-modal feature extraction network, used to extract features from the first image data and the second image data respectively to obtain the first modal features and the second modal features;

[0188] The single-modal feature extraction network includes M convolutional blocks, M pooling layers, and a multi-scale feature fusion module. The M pooling layers have different parameter settings and are used to process the features output by the M convolutional blocks into the same size. M is a positive integer greater than 1.

[0189] The first convolutional block among the M convolutional blocks is used to perform convolution processing on the input image data. The first pooling layer among the M pooling layers is used to perform pooling processing on the features output by the M convolutional layers respectively. The i-th convolutional block among the M convolutional blocks is used to perform convolution processing on the pooled features output by the (i-1)-th convolutional block among the M convolutional blocks. The j-th pooling layer among the M pooling layers is used to perform pooling processing on the features output by the (j-1)-th convolutional block among the M convolutional blocks. The multi-scale feature fusion module is used to fuse the target convolutional feature and the features output by the second to the M-th pooling layers among the M pooling layers. The target convolutional feature is the feature output by the first pooling layer on the M-th convolutional block among the M pooling layers, where i and j are both integers greater than 1 and less than or equal to M.

[0190] Optionally, the facial recognition device 800 also includes:

[0191] The Spatial Attention Vectorization (SAV) module is used to convert the first modality integrated feature, the second modality integrated feature, and the first fusion feature into feature vectors respectively, to obtain the first modality integrated feature vector, the second modality integrated feature vector, and the first fusion feature vector. The SAV module is used to perform weighted processing on the features of each channel in the input features, and to flatten the weighted features to obtain the feature vector corresponding to the input features.

[0192] The second feature fusion module 804 is used to fuse the first modality integrated feature vector, the second modality integrated feature vector, and the first fused feature vector to obtain the second fused feature vector;

[0193] The determining module 805 is used to determine the face recognition result based on the second fused feature vector.

[0194] Optionally, the SAV module includes K sets of convolutional layers and flattening layers, where K is equal to the number of channels of the input features;

[0195] The K sets of convolutional layers are used to perform convolution processing on the features of each channel in the input features, and the flattening layer is used to flatten the features output by the K sets of convolutional layers to obtain the feature vector corresponding to the input features.

[0196] Optionally, the integration module 803 is a single-modal feature integration module, used to integrate features of different scales of the first modal feature and features of different scales of the second modal feature to obtain the first modal integrated feature and the second modal integrated feature, wherein the single-modal feature integration module includes a third convolutional block.

[0197] The face recognition device 800 provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0198] The face recognition device 800 of this application embodiment extracts features from first image data and second image data to obtain first modality features and second modality features, wherein the first image data and the second image data are image data of different modalities of the face, respectively. The first modality features and the second modality features are fused to obtain a first fused feature. Features of different scales of the first modality features and features of different scales of the second modality features are integrated to obtain a first modality integrated feature and a second modality integrated feature. The first modality integrated feature, the second modality integrated feature, and the first fused feature are fused to obtain a second fused feature. The face recognition result is determined based on the second fused feature. Thus, by performing two-stage fusion of image data of different modalities of the face at the feature layer, complementary information between different modalities can be fully utilized and mined, achieving a deeper level of fusion of multimodal image data, thereby improving the face recognition effect.

[0199] This application also provides an electronic device. Since the principle by which the electronic device solves the problem is similar to the face recognition method in this application, the implementation of this electronic device can be found in the implementation of the method, and repeated details will not be described again. Figure 9 As shown, the electronic device according to an embodiment of this application includes:

[0200] Processor 900 is used to read the program from memory 920 and execute the following procedures:

[0201] Feature extraction is performed on the first image data and the second image data respectively to obtain the first modality feature and the second modality feature, wherein the first image data and the second image data are image data of different modalities of the face;

[0202] The first modal feature and the second modal feature are fused to obtain the first fused feature;

[0203] The features of the first modality at different scales and the features of the second modality at different scales are integrated to obtain the first modality integrated feature and the second modality integrated feature.

[0204] The first modality integration feature, the second modality integration feature, and the first fusion feature are fused together to obtain the second fusion feature;

[0205] The face recognition result is determined based on the second fusion feature.

[0206] Transceiver 910 is used to receive and send data under the control of processor 900.

[0207] Among them, Figure 9 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 900) and memory (memory 920). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface. Processor 900 is responsible for managing the bus architecture and general processing, and memory 920 can store data used by processor 900 during operation.

[0208] Optionally, the processor 900 is also used to read the program from the memory 920 and perform the following steps:

[0209] The first modal feature and the second modal feature are concatenated to obtain a bimodal feature;

[0210] The features of different channels in the dual-modal features are processed by convolution with N convolution branches respectively to obtain N convolution results. Each of the N convolution branches has the same structure, and N is a positive integer greater than 1.

[0211] The N convolution results are integrated to obtain the first fused feature.

[0212] Optionally, the N convolutional branches include a first convolutional block and N sets of second convolutional blocks, wherein the kernel sizes of the first convolutional block and the second convolutional blocks are different;

[0213] Processor 900 is also used to read programs from memory 920 and perform the following steps:

[0214] The first convolutional block is used to perform convolution processing on the dual-modal features to output the first convolutional features;

[0215] The first convolutional features are processed by grouping convolution through the N groups of second convolutional blocks respectively, and the N convolutional results are output.

[0216] Optionally, the processor 900 is also used to read the program from the memory 920 and perform the following steps:

[0217] A single-modal feature extraction network is used to extract features from the first image data and the second image data respectively to obtain the first modal features and the second modal features;

[0218] The single-modal feature extraction network includes M convolutional blocks, M pooling layers, and a multi-scale feature fusion module. The M pooling layers have different parameter settings and are used to process the features output by the M convolutional blocks into the same size. M is a positive integer greater than 1.

[0219] The first convolutional block among the M convolutional blocks is used to perform convolution processing on the input image data. The first pooling layer among the M pooling layers is used to perform pooling processing on the features output by the M convolutional layers respectively. The i-th convolutional block among the M convolutional blocks is used to perform convolution processing on the pooled features output by the (i-1)-th convolutional block among the M convolutional blocks. The j-th pooling layer among the M pooling layers is used to perform pooling processing on the features output by the (j-1)-th convolutional block among the M convolutional blocks. The multi-scale feature fusion module is used to fuse the target convolutional feature and the features output by the second to the M-th pooling layers among the M pooling layers. The target convolutional feature is the feature output by the first pooling layer on the M-th convolutional block among the M pooling layers, where i and j are both integers greater than 1 and less than or equal to M.

[0220] Optionally, the processor 900 is also used to read the program from the memory 920 and perform the following steps:

[0221] The first modality integrated feature, the second modality integrated feature, and the first fusion feature are converted into feature vectors using the Spatial Attention Vectorization (SAV) module, respectively, to obtain the first modality integrated feature vector, the second modality integrated feature vector, and the first fusion feature vector. The SAV module is used to perform weighted processing on the features of each channel in the input features and flatten the weighted features to obtain the feature vector corresponding to the input features.

[0222] The first modality integrated feature vector, the second modality integrated feature vector, and the first fused feature vector are fused to obtain the second fused feature vector;

[0223] The face recognition result is determined based on the second fused feature vector.

[0224] Optionally, the SAV module includes K sets of convolutional layers and flattening layers, where K is equal to the number of channels of the input features;

[0225] The K sets of convolutional layers are used to perform convolution processing on the features of each channel in the input features, and the flattening layer is used to flatten the features output by the K sets of convolutional layers to obtain the feature vector corresponding to the input features.

[0226] Optionally, the processor 900 is also used to read the program from the memory 920 and perform the following steps:

[0227] The first modality feature and the second modality feature are integrated by a single modality feature integration module to obtain the first modality integrated feature and the second modality integrated feature. The single modality feature integration module includes a third convolutional block.

[0228] The electronic device provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0229] Furthermore, the computer-readable storage medium of this application embodiment is used to store a computer program, which can be executed by a processor. Figure 1 The steps in the method embodiment shown.

[0230] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0231] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0232] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0233] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A face recognition method, characterized in that, include: Feature extraction is performed on the first image data and the second image data respectively to obtain the first modality feature and the second modality feature, wherein the first image data and the second image data are image data of different modalities of the face; The first modal feature and the second modal feature are fused to obtain the first fused feature; The features of the first modality at different scales and the features of the second modality at different scales are integrated to obtain the first modality integrated feature and the second modality integrated feature. The first modality integration feature, the second modality integration feature, and the first fusion feature are fused together to obtain the second fusion feature; The face recognition result is determined based on the second fusion feature; The step of extracting features from the first image data and the second image data respectively to obtain first modality features and second modality features includes: A single-modal feature extraction network is used to extract features from the first image data and the second image data respectively to obtain the first modal features and the second modal features; The single-modal feature extraction network includes M convolutional blocks, M pooling layers, and a multi-scale feature fusion module. The M pooling layers have different parameter settings and are used to process the features output by the M convolutional blocks into the same size. M is a positive integer greater than 1. The first convolutional block among the M convolutional blocks is used to perform convolution processing on the input image data. The first pooling layer among the M pooling layers is used to perform pooling processing on the features output by the M convolutional blocks respectively. The i-th convolutional block among the M convolutional blocks is used to perform convolution processing on the pooled features output by the (i-1)-th convolutional block among the M convolutional blocks. The j-th pooling layer among the M pooling layers is used to perform pooling processing on the features output by the (j-1)-th convolutional block among the M convolutional blocks. The multi-scale feature fusion module is used to fuse the target convolutional feature and the features output by the second to the M-th pooling layers among the M pooling layers. The target convolutional feature is the feature output by the first pooling layer on the M-th convolutional block among the M pooling layers. i and j are both integers greater than 1 and less than or equal to M.

2. The method according to claim 1, characterized in that, The process of fusing the first modal feature and the second modal feature to obtain the first fused feature includes: The first modal feature and the second modal feature are concatenated to obtain a bimodal feature; The features of different channels in the dual-modal features are processed by convolution with N convolution branches respectively to obtain N convolution results. Each of the N convolution branches has the same structure, and N is a positive integer greater than 1. The N convolution results are integrated to obtain the first fused feature.

3. The method according to claim 2, characterized in that, The N convolutional branches include a first convolutional block and N sets of second convolutional blocks, wherein the kernel sizes of the first convolutional block and the second convolutional blocks are different; The process involves using N convolutional branches to perform convolutional processing on features from different channels in the dual-modal features, resulting in N convolutional results, including: The first convolutional block is used to perform convolution processing on the dual-modal features to output the first convolutional features; The first convolutional features are processed by grouping convolution through the N groups of second convolutional blocks respectively, and the N convolutional results are output.

4. The method according to any one of claims 1 to 3, characterized in that, After integrating the features of different scales of the first modality feature and the features of different scales of the second modality feature respectively, and before fusing the first modality integrated feature, the second modality integrated feature, and the first fused feature, the method further includes: The first modality integrated feature, the second modality integrated feature, and the first fusion feature are converted into feature vectors using the Spatial Attention Vectorization (SAV) module, respectively, to obtain the first modality integrated feature vector, the second modality integrated feature vector, and the first fusion feature vector. The SAV module is used to perform weighted processing on the features of each channel in the input features and flatten the weighted features to obtain the feature vector corresponding to the input features. The process of fusing the first modality integration feature, the second modality integration feature, and the first fusion feature to obtain the second fusion feature includes: The first modality integrated feature vector, the second modality integrated feature vector, and the first fused feature vector are fused to obtain the second fused feature vector; The step of determining the face recognition result based on the second fusion feature includes: The face recognition result is determined based on the second fused feature vector.

5. The method according to claim 4, characterized in that, The SAV module includes K sets of convolutional layers and flattening layers, where K is equal to the number of channels of the input features. The K sets of convolutional layers are used to perform convolution processing on the features of each channel in the input features, and the flattening layers are used to flatten the features output by the K sets of convolutional layers to obtain the feature vector corresponding to the input features.

6. The method according to any one of claims 1 to 3, characterized in that, The process of integrating features of different scales of the first modality feature and features of different scales of the second modality feature to obtain the first modality integrated feature and the second modality integrated feature includes: The first modality feature and the second modality feature are integrated by a single modality feature integration module to obtain the first modality integrated feature and the second modality integrated feature. The single modality feature integration module includes a third convolutional block.

7. A face recognition device, characterized in that, include: The feature extraction module is used to extract features from the first image data and the second image data respectively to obtain first modality features and second modality features, wherein the first image data and the second image data are image data of different modalities of the face; The first feature fusion module is used to fuse the first modal feature and the second modal feature to obtain the first fused feature; An integration module is used to integrate features of different scales of the first modality feature and features of different scales of the second modality feature to obtain the first modality integrated feature and the second modality integrated feature. The second feature fusion module is used to fuse the first modality integrated feature, the second modality integrated feature, and the first fusion feature to obtain the second fusion feature; The determining module is used to determine the face recognition result based on the second fused feature; The feature extraction module is a single-modal feature extraction network, used to extract features from the first image data and the second image data respectively, to obtain the first modal features and the second modal features; The single-modal feature extraction network includes M convolutional blocks, M pooling layers, and a multi-scale feature fusion module. The M pooling layers have different parameter settings and are used to process the features output by the M convolutional blocks into the same size. M is a positive integer greater than 1. The first convolutional block among the M convolutional blocks is used to perform convolution processing on the input image data. The first pooling layer among the M pooling layers is used to perform pooling processing on the features output by the M convolutional blocks respectively. The i-th convolutional block among the M convolutional blocks is used to perform convolution processing on the pooled features output by the (i-1)-th convolutional block among the M convolutional blocks. The j-th pooling layer among the M pooling layers is used to perform pooling processing on the features output by the (j-1)-th convolutional block among the M convolutional blocks. The multi-scale feature fusion module is used to fuse the target convolutional feature and the features output by the second to the M-th pooling layers among the M pooling layers. The target convolutional feature is the feature output by the first pooling layer on the M-th convolutional block among the M pooling layers. i and j are both integers greater than 1 and less than or equal to M.

8. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program in the memory to implement the steps of the face recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the face recognition method as described in any one of claims 1 to 6.

Images