Dual-mode palm recognition method based on dual-stream deep learning recognition network
Patent Information
- Application Number
- CN202611153098.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-31
- Publication Date
- 2026-10-13
AI Technical Summary
[0007]本发明的目的在于克服现有技术中双模态识别特征融合不充分、环境噪声干扰严重以及多模态特征分布不一致等缺点，提出了一种基于双流深度学习识别网络的双模态手掌识别方法，通过频域精炼与自适应动态融合，实现可见光与近红外模态特征的深度集成，在提升识别精度的同时增强对复杂环境的鲁棒性
[0041]1、显著增强了复杂环境下的识别鲁棒性。
Smart Images
Figure CN122906761A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical fields of biometric recognition and computer vision, and in particular to a bimodal palm recognition method based on a dual-stream deep learning recognition network, which can be applied to scenarios requiring high-precision identity verification such as identity authentication, access control and security, and financial payment. Background Technology
[0002] Most publicly available dual-modal palm recognition schemes currently employ symmetric dual-stream convolutional networks to extract features from both modalities separately, and then achieve feature interaction through simple concatenation and additive fusion. However, these schemes suffer from the following core shortcomings in practical applications:
[0003] 1. Poor robustness to environmental interference: Existing solutions mostly perform feature extraction in the spatial domain, which cannot effectively filter out invalid interference caused by changes in lighting, motion blur, and equipment noise. Noise information will continue to accumulate in the deep layers of the network, resulting in insufficient purity of feature discrimination.
[0004] 2. Insufficient flexibility in modal fusion: Most fusion methods use fixed weights, which cannot dynamically adjust the contribution of each modality according to the quality fluctuations of the dual-modal images. When a modality is overexposed, occluded, or suffers quality degradation, it will seriously affect the final recognition effect.
[0005] 3. Insufficient utilization of multi-scale features: Existing solutions mostly only fuse deep semantic features at the end of the network, ignoring the local texture and subtle anatomical structure information contained in the mid-level features. Continuous downsampling of deep networks will lead to the loss of key details and reduce the stability of cross-scene recognition.
[0006] 4. The cross-modal representation gap is difficult to eliminate: Most networks use a single classification loss for training, which cannot effectively bridge the feature distance of the same identity in different modalities. The intra-class distance is large, the inter-class discrimination is insufficient, and the cross-modal matching accuracy is limited. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of existing technologies, such as insufficient fusion of bimodal recognition features, severe environmental noise interference, and inconsistent distribution of multimodal features. This invention proposes a bimodal palm recognition method based on a dual-stream deep learning recognition network. Through frequency domain refinement and adaptive dynamic fusion, it achieves deep integration of visible light and near-infrared modal features, thereby improving recognition accuracy and enhancing robustness to complex environments.
[0008] To achieve the above objectives, the technical solution provided by this invention is: a dual-modal palm recognition method based on a dual-stream deep learning recognition network, wherein the dual-stream deep learning recognition network includes: The symmetrical dual-stream feature extraction pathway, which belongs to the input-end feature parsing channel, learns the shallow texture of visible light images and the subcutaneous vein structure of near-infrared images through two sets of parameter-independent deep convolution branches, and outputs feature maps of various modalities. The spectrum discrimination and refinement module, referred to as SDRM, is a mid-level feature frequency domain purification component. It maps mid-level features to the frequency domain through a two-dimensional real number fast Fourier transform and uses a learnable complex weight matrix to adjust the frequency gain in order to suppress low-frequency illumination and high-frequency noise while retaining mid-frequency ridge features, and outputs a mid-level refined feature map. The adaptive modality selection module, referred to as SK-Fusion, is a deep feature dynamic fusion component. It gathers global information of deep abstract features through parallel multi-scale kernel perception branches and calculates adaptive complementary weights in the channel and spatial dimensions to achieve dynamic fusion of dual modal features and output a deep fusion feature map. The pyramid feature fusion architecture belongs to the multi-scale feature alignment integration network. It performs spatial downsampling on the mid-level refined feature map output by the SDRM through convolutional layers, so that the output feature map is forcibly aligned with the resolution of the deep fusion feature output by the SK-Fusion and channel concatenation is performed to construct a multi-scale fusion feature map in a cross-scale integration manner. The dual-stream sensing module, referred to as DSPM, is a topological spatial feature compression component. It extracts local texture and macroscopic topological relationships through an asymmetric parallel sensing structure composed of standard convolutional streams and dilated convolutional streams. It also anchors key regions by combining spatial and channel dual attention mechanisms and compresses them into fixed-dimensional discriminative feature vectors through weighted residual skip-connection dimensionality reduction. The joint loss function constraint architecture, belonging to the feedback backpropagation training constraint supervision network, is used to simultaneously calculate the angle discrimination loss with additive angle cosine margin, the contrast similarity loss for constructing cross-modal sample pairs, and the manifold consistency loss for constraining the overlap of cross-modal manifold spaces during the network training phase. Through a multi-task joint driving approach, it guides the dual-stream deep learning recognition network to simultaneously compress intra-class distance and expand inter-class distance in the hypersphere and low-dimensional manifold spaces.
[0009] The specific implementation of the dual-modal palm recognition method includes the following steps:
[0010] 1) Obtain bimodal palm images of individuals with different identities and preprocess them to obtain spatially aligned visible light and near-infrared images of each individual, which will be used as the training set for subsequent network training; wherein, the bimodal palm image of each individual includes a visible light palm print image and a near-infrared palm vein image.
[0011] 2) The training set is fed into the dual-stream deep learning recognition network for iterative training. During training, the visible light image and near-infrared image are first input into the symmetric dual-stream feature extraction path for deep semantic modeling, outputting corresponding modal feature maps at each level. Subsequently, the obtained mid-level modal feature maps are fed into the SDRM for frequency domain discrimination and purification, outputting mid-level refined feature maps. At the same time, the obtained deep modal feature maps are fed into the SK-Fusion for dynamic weight fusion, generating deep fused feature maps. Next, the mid-level refined feature maps and the deep fused feature maps are fed into the pyramid feature fusion architecture for spatial alignment and channel concatenation, constructing multi-scale fused feature maps, which are then input into the DSPM for spatial topology extraction and spatial channel calibration, reducing dimensionality and compressing them into fixed-dimensional discriminative feature vectors. Finally, the joint loss function constraint architecture is used to calculate the joint loss value between the prediction result of the dual-stream deep learning recognition network and the real label. Based on the joint loss value, the parameters of the dual-stream deep learning recognition network are iteratively optimized using the backpropagation algorithm to obtain the trained dual-stream deep learning recognition network.
[0012] 3) Input the bimodal palm image of the individual to be identified into the trained dual-stream deep learning recognition network. The network automatically extracts the discriminative feature vector of the palm and outputs the identity recognition result of the palm through feature comparison, thus completing the bimodal palm recognition task.
[0013] Further, in step 2), the symmetric dual-stream feature extraction pathway uses EfficientNet-B0 based on the moving-flipping bottleneck convolution MBConv as the basic unit. It learns the surface texture of visible light images and the subcutaneous vein structure of near-infrared images through parameter-independent dual-stream branches. The mid-level modal feature map is extracted from the fourth stage of EfficientNet-B0 and is divided into visible light mid-level features and near-infrared mid-level features according to the feature type. Taking advantage of the fact that this stage is at the receptive field expansion node, it preserves local spatial texture and extracts preliminary semantic features. The deep modal feature map is extracted from the end of EfficientNet-B0 and is used to capture highly abstract global identity representations. Together with the mid-level modal feature map, it constitutes a multi-level feature system covering details, structure, and semantics.
[0014] Furthermore, in step 2), the specific processing steps of the SDRM are as follows:
[0015] Input mid-level modal feature map Mapping to a high-dimensional space, where R represents the real number field, and C, H, and W represent the number of channels, image height, and image width, respectively, yields the feature map. ;Utilize depthwise separable convolutions to pair feature maps After performing spatial local modeling, it is split into two parallel branches along the channel dimension. and Using the GELU nonlinear activation function to Process and with Element-wise multiplication yields the gated enhanced feature X. gate Then, linear projection is used to restore the number of channels, thus obtaining the preprocessed features. ;
[0016] Using rearrangement operators to preprocess features Divided into several sizes Local feature block X patch For each local feature block X patch Perform a real-number fast Fourier transform (rFFT2) to map the frequency from the spatial domain to the frequency domain, obtaining the spectral components containing complex information. ;
[0017] Construct a learnable complex weight matrix with the same dimension as the spectral components as the frequency discrimination matrix. ;Will and Performing pointwise Hadamard product operations enables adaptive gain control of different frequency components in the spectrum. This adaptive gain control is achieved through... Automatic iterative updates preserve the mid-frequency response that characterizes the palm print ridge features and suppress low-frequency illumination components and high-frequency noise components.
[0018] Using Inverse Fast Fourier Transform (irFFT2) to extract spectral components Reverting to the spatial domain yields local feature blocks with improved discrimination purity. Finally, the rearrangement operator is used to handle all local feature blocks. Reorganize according to the original spatial location to output the final mid-level refined feature map.
[0019] Furthermore, in step 2), the specific processing steps of SK-Fusion are as follows:
[0020] Channel adjustment is performed on the visible light and near-infrared deep modal feature maps using convolution operators, followed by element-wise addition and fusion to obtain global descriptive features with preliminary dual-modal interaction information. The visible light and near-infrared deep modal feature maps are simultaneously fed into two sets of deep convolution branches with different receptive fields:
[0021] Small kernel branch: Employs 3×3 depthwise convolution to extract fine local feature maps;
[0022] Large kernel branch: Following the previous convolutional layer, a 5×5 dilated convolution is used to further expand the receptive field and extract a wider range of spatial context semantics;
[0023] Global average pooling (GAP) is performed on the global descriptive features to obtain channel statistics. Then, a bottleneck layer containing two fully connected layers is used for nonlinear mapping to extract higher-order modality association weight vectors. Specifically, the features of two sets of deep convolutional branches are concatenated using a connection operator, and two dimensions of statistics are extracted simultaneously: first, average pooling and max pooling are performed on the concatenated features to generate a dual-channel spatial descriptor agg; second, global average pooling (GAP) is performed on the concatenated features, and then the features are compressed to the latent space dimension d through a bottleneck structure consisting of 1×1 convolutions, normalization layers, and ReLU activation functions. , It is the vector dimension before compression. It's the compression ratio. It sets a minimum dimension threshold;
[0024] A 1×1 convolutional layer is used to increase the dimensionality of the latent space features, and Softmax normalization is performed to generate mutually exclusive and complementary adaptive weight coefficients in the modality and kernel dimensions. and The weighting coefficient , The reshape operation is combined element-wise with the spatial descriptor agg to generate the final dynamic selection operator that acts on each branch. and ;
[0025] Using the dynamic selection operator and The original features of the two sets of deep convolutional branches are weighted and summed pixel by pixel to obtain the fused intermediate features. Finally, the enhanced deep fused feature map is output through a 1×1 convolutional layer and a Sigmoid activation function.
[0026] Furthermore, in step 2), the specific processing steps of the pyramid feature fusion architecture are as follows:
[0027] Spatially downsample the mid-level refined feature map after SDRM processing using a convolutional layer with a stride of 2, so that its spatial resolution is forcibly aligned with the deep fusion feature map output by SK-Fusion.
[0028] By using a concatenation operator, the aligned visible light mid-layer features, near-infrared mid-layer features, and deep fused feature maps are cascaded and stacked along the channel dimension to construct a multi-scale fused feature map. .
[0029] Furthermore, in step 2), the specific processing steps of the DSPM are as follows:
[0030] Input multi-scale fused feature map The input is synchronously fed into two parallel branches that compute features differently, in order to achieve the collaborative extraction of local details and macroscopic topology. The specific details of these parallel branches are as follows:
[0031] Standard Convolutional Flow: Employs two cascaded 3×3 convolutional layers with a dilation rate of 1 to extract fine yet continuous skin texture and ridge features from the multi-scale fused feature map, outputting a local feature map. ;
[0032] Dilated Convolutional Flow: Employs two asymmetric dilated convolutional layers. The dilation rate of the first dilated convolutional layer is set to 2, and that of the second dilated convolutional layer is set to 3. By increasing the sampling stride, a larger receptive field is obtained to capture the cross-regional geometric topology of the palm and the relative spatial positions between feature points, outputting a topological feature map. ;
[0033] Using the join operator and Channel cascading is performed to obtain the cascaded feature map. And using 1×1 convolutional layers to Channel compression is performed to achieve initial intermodal interaction, and the compressed data is then processed. The input undergoes a bottleneck structure consisting of two 1×1 convolutional layers, which is then nonlinearly mapped to generate channel weight vectors. These vectors are then processed sequentially using a channel attention module (CA) and a spatial attention module (SA) to generate feature maps. ;
[0034] In addition, a perceptual enhancement path parallel to the aforementioned standard convolutional flow and dilated convolutional flow is introduced, along with predefined hyperparameters. As the gain coefficient of the perception enhancement path, it is calculated using the formula... The features in the formula are integrated, where The input features are aligned in dimension after a 1×1 convolution. This represents the enhanced features after processing by the collaborative attention mechanism; To control the balance factor of the enhanced feature ratio, which is used to dynamically adjust the linear fusion ratio of the feature streams of the perception enhancement path, the final fixed-dimensional discriminative feature vector is obtained by performing ReLU nonlinear activation. .
[0035] Furthermore, in step 2), the total loss of the joint loss function constructed by the joint loss function constraint architecture is... The calculation formula is: ,in, To determine the loss based on the angle; To compare similarity loss, For manifold consistency loss, , This is the preset balance coefficient;
[0036] The angle discrimination loss By introducing an additive angle cosine margin at the angle between the feature vector and the weight vector, the features are mapped to a hyperspherical space to construct a strict angle decision boundary, thereby compressing the intra-class distance and expanding the inter-class distance.
[0037] The comparison similarity loss By constructing cross-modal positive and negative sample pairs, the following can be achieved: using associated labels to narrow the feature mapping distance of the same identity in visible light images and near-infrared images, while widening the modal feature distance between individuals with different identities, thus explicitly eliminating the identity representation gap between modalities;
[0038] The manifold consistency loss The mean square error operator is used to constrain the visible light and near-infrared modal features of the same individual to maintain a high degree of overlap in the manifold space, ensuring that the network has manifold stability against the unique noise under different modes.
[0039] Furthermore, the balance coefficient , The balance coefficient is heuristically adjusted based on the magnitude of the loss during the training phase. The value range is 0.1 to 0.5. The value range is 0.01 to 0.1.
[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0041] 1. Significantly enhances the robustness of recognition in complex environments.
[0042] By introducing a spectral discrimination refinement module (SDRM), this invention enables frequency domain decoupling of image features at the physical level. Utilizing a learnable frequency discrimination matrix, it effectively filters out illuminance fluctuation interference in the visible light mode and discrete electronic noise in the near-infrared mode, accurately preserving and enhancing the mid-frequency features of palm print ridges and wrinkles. This solves the problem of significant reduction in recognition rate under strong light or low-quality imaging conditions in traditional methods.
[0043] 2. Adaptive deep fusion of multimodal information has been achieved.
[0044] The adaptive modality selection module (SK-Fusion) designed in this invention can dynamically adjust weights based on the quality of the input image. Through a multi-scale kernel perception branch, it can perceive changes in the recognition environment in real time and automatically increase the contribution of higher-quality modalities, overcoming the limitation of traditional fixed-weight fusion methods that cannot take into account the dynamic differences between different modalities.
[0045] 3. Improved the accuracy of expressing spatial topological relationships.
[0046] The dual-stream sensing module (DSPM) employs a parallel design of standard convolutional and dilated convolutional streams, simultaneously capturing both local texture details and macroscopic anatomical structures of palm prints. Combined with channel attention and spatial attention mechanisms, it automatically anchors key discrimination regions such as palm lines and suppresses invalid interference from palm edges and background areas, resulting in generated fixed-dimensional discriminative feature vectors with extremely high spatial discriminative power.
[0047] 4. Effectively eliminates the cross-modal identity representation gap.
[0048] This invention constructs a joint constraint mechanism that incorporates manifold consistency loss, thereby forcibly constraining the visible light and near-infrared features of individuals with the same identity to remain consistent in the manifold space at the feature distribution level. Combined with angular discrimination loss, it further compresses intra-class distances and expands inter-class decision boundaries, solving the technical challenge of inconsistent multimodal feature distributions caused by different imaging mechanisms.
[0049] 5. It has good engineering applicability and scalability.
[0050] This invention employs a symmetrical dual-stream lightweight architecture for a dual-stream deep learning recognition network. While maintaining recognition accuracy, it significantly reduces the number of network parameters through parameter sharing and efficient convolution operators. The end-to-end design of this invention allows for easy deployment in embedded handheld recognition terminals, meeting the needs of high real-time applications such as financial payments and security attendance systems. Attached Figure Description
[0051] Figure 1 This is a structural diagram of the dual-stream deep learning recognition network in the method of this invention.
[0052] Figure 2 This is a structural diagram of the spectrum discrimination and refining module (referred to as SDRM) in the method of the present invention.
[0053] Figure 3 This is a structural diagram of the adaptive mode selection module (referred to as SK-Fusion) in the method of the present invention.
[0054] Figure 4 This is a structural diagram of the dual-stream sensing module (referred to as DSPM) in the method of the present invention.
[0055] Figure 5 This is a structural diagram of the joint loss function constraint architecture in the method of the present invention.
[0056] Figure 6 This is a schematic diagram of the ROI region image after standardization according to the method of the present invention.
[0057] Figure 7This is a flowchart of the identification and matching process of the present invention. Detailed Implementation
[0058] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.
[0059] like Figures 1 to 7 As shown, this embodiment discloses a dual-modal palm recognition method based on a dual-stream deep learning recognition network, wherein the dual-stream deep learning recognition network includes: The symmetric dual-stream feature extraction pathway, belonging to the input-side feature parsing channel, learns features through two sets of parameter-independent depthwise convolutional branches, such as... Figure 6 The system displays the superficial texture of visible light images and the subcutaneous venous structure of near-infrared images, and outputs feature maps of various modalities. The spectrum discrimination and refinement module, referred to as SDRM, is a mid-level feature frequency domain purification component. It maps mid-level features to the frequency domain through a two-dimensional real number fast Fourier transform and uses a learnable complex weight matrix to adjust the frequency gain in order to suppress low-frequency illumination and high-frequency noise while retaining mid-frequency ridge features, and outputs a mid-level refined feature map. The adaptive modality selection module, referred to as SK-Fusion, is a deep feature dynamic fusion component. It gathers global information of deep abstract features through parallel multi-scale kernel perception branches and calculates adaptive complementary weights in the channel and spatial dimensions to achieve dynamic fusion of dual modal features and output a deep fusion feature map. The pyramid feature fusion architecture belongs to the multi-scale feature alignment integration network. It performs spatial downsampling on the mid-level refined feature map output by the SDRM through convolutional layers, so that the output feature map is forcibly aligned with the resolution of the deep fusion feature output by the SK-Fusion and channel concatenation is performed to construct a multi-scale fusion feature map in a cross-scale integration manner. The dual-stream sensing module, referred to as DSPM, is a topological spatial feature compression component. It extracts local texture and macroscopic topological relationships through an asymmetric parallel sensing structure composed of standard convolutional streams and dilated convolutional streams. It also anchors key regions by combining spatial and channel dual attention mechanisms and compresses them into fixed-dimensional discriminative feature vectors through weighted residual skip-connection dimensionality reduction. The joint loss function constraint architecture, belonging to the feedback backpropagation training constraint supervision network, is used to simultaneously calculate the angle discrimination loss with additive angle cosine margin, the contrast similarity loss for constructing cross-modal sample pairs, and the manifold consistency loss for constraining the overlap of cross-modal manifold spaces during the network training phase. Through a multi-task joint driving approach, it guides the dual-stream deep learning recognition network to simultaneously compress intra-class distance and expand inter-class distance in the hypersphere and low-dimensional manifold spaces.
[0060] The specific implementation of the dual-modal palm recognition method includes the following steps:
[0061] 1) Obtain bimodal palm images of individuals with different identities and preprocess them. Use geometric correction, affine transformation and region of interest extraction algorithms to process the original images, correct the palm rotation angle and uniformly crop them into normalized visible light images and near-infrared images of consistent size and spatial alignment, so as to obtain spatially aligned visible light images and near-infrared images for each individual, which are used as training sets for subsequent network training; wherein, the bimodal palm image of each individual includes a visible light palm print image and a near-infrared palm vein image;
[0062] 2) The training set is fed into the dual-stream deep learning recognition network for iterative training. During training, the visible light image and near-infrared image are first input into the symmetric dual-stream feature extraction path for deep semantic modeling, outputting corresponding modal feature maps at each level. Subsequently, the obtained mid-level modal feature maps are fed into the SDRM for frequency domain discrimination and purification, outputting mid-level refined feature maps. At the same time, the obtained deep modal feature maps are fed into the SK-Fusion for dynamic weight fusion, generating deep fused feature maps. Next, the mid-level refined feature maps and the deep fused feature maps are fed into the pyramid feature fusion architecture for spatial alignment and channel concatenation, constructing multi-scale fused feature maps, which are then input into the DSPM for spatial topology extraction and spatial channel calibration, reducing dimensionality and compressing them into fixed-dimensional discriminative feature vectors. Finally, the joint loss function constraint architecture is used to calculate the joint loss value between the prediction result of the dual-stream deep learning recognition network and the real label. Based on the joint loss value, the parameters of the dual-stream deep learning recognition network are iteratively optimized using the backpropagation algorithm to obtain the trained dual-stream deep learning recognition network.
[0063] 3) Input the bimodal palm image of the individual to be identified into the trained dual-stream deep learning recognition network. The network automatically extracts the discriminative feature vector of the palm and outputs the identity recognition result of the palm through feature comparison, thus completing the bimodal palm recognition task.
[0064] In a specific example, the symmetric dual-stream feature extraction pathway uses EfficientNet-B0 based on the moving-flipping bottleneck convolution MBConv as the basic unit. It learns the surface texture of visible light images and the subcutaneous vein structure of near-infrared images through parameter-independent dual-stream branches. The mid-level modal feature map is extracted from the fourth stage of EfficientNet-B0 and is divided into visible light mid-level features and near-infrared mid-level features according to feature type. Taking advantage of the fact that this stage is at the receptive field expansion node, it preserves local spatial texture and extracts preliminary semantic features. The deep modal feature map is extracted from the end of EfficientNet-B0 and is used to capture highly abstract global identity representation. Together with the mid-level modal feature map, it constitutes a multi-level feature system covering details, structure, and semantics.
[0065] In a specific example, the specific processing steps of the SDRM are as follows:
[0066] Input mid-level modal feature map Mapping to a high-dimensional space, where R represents the real number field, and C, H, and W represent the number of channels, image height, and image width, respectively, yields the feature map. ;Utilize depthwise separable convolutions to pair feature maps After performing spatial local modeling, it is split into two parallel branches along the channel dimension. and Using the GELU nonlinear activation function to Process and with Element-wise multiplication yields the gated enhanced feature X. gate Then, linear projection is used to restore the number of channels, thus obtaining the preprocessed features. ;
[0067] Using rearrangement operators to preprocess features Divided into several sizes Local feature block X patch For each local feature block X patch Perform a real-number fast Fourier transform (rFFT2) to map the frequency from the spatial domain to the frequency domain, obtaining the spectral components containing complex information. ;
[0068] Construct a learnable complex weight matrix with the same dimension as the spectral components as the frequency discrimination matrix. ;Will and Performing pointwise Hadamard product operations enables adaptive gain control of different frequency components in the spectrum. This adaptive gain control is achieved through... Automatic iterative updates preserve the mid-frequency response that characterizes the palm print ridge features and suppress low-frequency illumination components and high-frequency noise components.
[0069] Using Inverse Fast Fourier Transform (irFFT2) to extract spectral components Reverting to the spatial domain yields local feature blocks with improved discrimination purity. Finally, the rearrangement operator is used to handle all local feature blocks. Reorganize according to the original spatial location to output the final mid-level refined feature map.
[0070] In a specific example, the specific processing steps of SK-Fusion are as follows:
[0071] Channel adjustment is performed on the visible light and near-infrared deep modal feature maps using convolution operators, followed by element-wise addition and fusion to obtain global descriptive features with preliminary dual-modal interaction information. The visible light and near-infrared deep modal feature maps are simultaneously fed into two sets of deep convolution branches with different receptive fields:
[0072] Small kernel branch: Employs 3×3 depthwise convolution to extract fine local feature maps;
[0073] Large kernel branch: Following the previous convolutional layer, a 5×5 dilated convolution is used to further expand the receptive field and extract a wider range of spatial context semantics;
[0074] Global average pooling (GAP) is performed on the global descriptive features to obtain channel statistics. Then, a bottleneck layer containing two fully connected layers is used for nonlinear mapping to extract higher-order modality association weight vectors. Specifically, the features of two sets of deep convolutional branches are concatenated using a connection operator, and two dimensions of statistics are extracted simultaneously: first, average pooling and max pooling are performed on the concatenated features to generate a dual-channel spatial descriptor agg; second, global average pooling (GAP) is performed on the concatenated features, and then the features are compressed to the latent space dimension d through a bottleneck structure consisting of 1×1 convolutions, normalization layers, and ReLU activation functions. , It is the vector dimension before compression. It's the compression ratio. It sets a minimum dimension threshold;
[0075] A 1×1 convolutional layer is used to increase the dimensionality of the latent space features, and Softmax normalization is performed to generate mutually exclusive and complementary adaptive weight coefficients in the modality and kernel dimensions. and The weighting coefficient , The reshape operation is combined element-wise with the spatial descriptor agg to generate the final dynamic selection operator that acts on each branch. and ;
[0076] Using the dynamic selection operator and The original features of the two sets of deep convolutional branches are weighted and summed pixel by pixel to obtain the fused intermediate features. Finally, the enhanced deep fused feature map is output through a 1×1 convolutional layer and a Sigmoid activation function.
[0077] In a specific example, the specific processing steps of the pyramid feature fusion architecture are as follows:
[0078] Spatially downsample the mid-level refined feature map after SDRM processing using a convolutional layer with a stride of 2, so that its spatial resolution is forcibly aligned with the deep fusion feature map output by SK-Fusion.
[0079] By using a concatenation operator, the aligned visible light mid-layer features, near-infrared mid-layer features, and deep fused feature maps are cascaded and stacked along the channel dimension to construct a multi-scale fused feature map. .
[0080] In a specific example, the specific processing steps of the DSPM are as follows:
[0081] Input multi-scale fused feature map The input is synchronously fed into two parallel branches that compute features differently, in order to achieve the collaborative extraction of local details and macroscopic topology. The specific details of these parallel branches are as follows:
[0082] Standard Convolutional Flow: Employs two cascaded 3×3 convolutional layers with a dilation rate of 1 to extract fine yet continuous skin texture and ridge features from the multi-scale fused feature map, outputting a local feature map. ;
[0083] Dilated Convolutional Flow: Employs two asymmetric dilated convolutional layers. The dilation rate of the first dilated convolutional layer is set to 2, and that of the second dilated convolutional layer is set to 3. By increasing the sampling stride, a larger receptive field is obtained to capture the cross-regional geometric topology of the palm and the relative spatial positions between feature points, outputting a topological feature map. ;
[0084] Using the join operator and Channel cascading is performed to obtain the cascaded feature map. And using 1×1 convolutional layers to Channel compression is performed to achieve initial intermodal interaction, and the compressed data is then processed. The input undergoes a bottleneck structure consisting of two 1×1 convolutional layers, which is then nonlinearly mapped to generate channel weight vectors. These vectors are then processed sequentially using a channel attention module (CA) and a spatial attention module (SA) to generate feature maps. ;
[0085] In addition, a perceptual enhancement path parallel to the aforementioned standard convolutional flow and dilated convolutional flow is introduced, along with predefined hyperparameters. As the gain coefficient of the perception enhancement path, it is calculated using the formula... The features in the formula are integrated, where The input features are aligned in dimension after a 1×1 convolution. This represents the enhanced features after processing by the collaborative attention mechanism; To control the balance factor of the enhanced feature ratio, which is used to dynamically adjust the linear fusion ratio of the feature streams of the perception enhancement path, the final fixed-dimensional discriminative feature vector is obtained by performing ReLU nonlinear activation. .
[0086] In a specific example, the total loss of the joint loss function constructed by the joint loss function constraint architecture. The calculation formula is: ,in, To determine the loss based on the angle; To compare similarity loss, For manifold consistency loss, , This is the preset balance coefficient;
[0087] The angle discrimination loss By introducing an additive angle cosine margin at the angle between the feature vector and the weight vector, the features are mapped to a hyperspherical space to construct a strict angle decision boundary, thereby compressing the intra-class distance and expanding the inter-class distance.
[0088] The comparison similarity loss By constructing cross-modal positive and negative sample pairs, the following can be achieved: using associated labels to narrow the feature mapping distance of the same identity in visible light images and near-infrared images, while widening the modal feature distance between individuals with different identities, thus explicitly eliminating the identity representation gap between modalities;
[0089] The manifold consistency loss The mean square error operator is used to constrain the visible light and near-infrared modal features of the same individual to maintain a high degree of overlap in the manifold space, ensuring that the network has manifold stability against the unique noise under different modes;
[0090] The balance coefficient , The balance coefficient is heuristically adjusted based on the magnitude of the loss during the training phase. The value range is 0.1 to 0.5. The value range is 0.01 to 0.1.
[0091] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A dual-modal palm recognition method based on a dual-stream deep learning recognition network, characterized in that, The dual-stream deep learning recognition network includes: The symmetrical dual-stream feature extraction pathway, which belongs to the input-end feature parsing channel, learns the shallow texture of visible light images and the subcutaneous vein structure of near-infrared images through two sets of parameter-independent deep convolution branches, and outputs feature maps of various modalities. The spectrum discrimination and refinement module, referred to as SDRM, is a mid-level feature frequency domain purification component. It maps mid-level features to the frequency domain through a two-dimensional real number fast Fourier transform and uses a learnable complex weight matrix to adjust the frequency gain in order to suppress low-frequency illumination and high-frequency noise while retaining mid-frequency ridge features, and outputs a mid-level refined feature map. The adaptive modality selection module, referred to as SK-Fusion, is a deep feature dynamic fusion component. It gathers global information of deep abstract features through parallel multi-scale kernel perception branches and calculates adaptive complementary weights in the channel and spatial dimensions to achieve dynamic fusion of dual modal features and output a deep fusion feature map. The pyramid feature fusion architecture belongs to the multi-scale feature alignment integration network. It performs spatial downsampling on the mid-level refined feature map output by the SDRM through convolutional layers, so that the output feature map is forcibly aligned with the resolution of the deep fusion feature output by the SK-Fusion and channel concatenation is performed to construct a multi-scale fusion feature map in a cross-scale integration manner. The dual-stream sensing module, referred to as DSPM, is a topological spatial feature compression component. It extracts local texture and macroscopic topological relationships through an asymmetric parallel sensing structure composed of standard convolutional streams and dilated convolutional streams. It also anchors key regions by combining spatial and channel dual attention mechanisms and compresses them into fixed-dimensional discriminative feature vectors through weighted residual skip-connection dimensionality reduction. The joint loss function constraint architecture, which belongs to the feedback backpropagation training constraint supervision network, is used to simultaneously calculate the angle discrimination loss with additive angle cosine margin, the contrast similarity loss for constructing cross-modal sample pairs, and the manifold consistency loss for constraining the overlap of cross-modal manifold space during the network training phase. Through a multi-task joint driving approach, it guides the dual-stream deep learning recognition network to simultaneously compress intra-class distance and expand inter-class distance in the hypersphere and low-dimensional manifold space. The specific implementation of the dual-modal palm recognition method includes the following steps: 1) Obtain bimodal palm images of individuals with different identities and preprocess them to obtain spatially aligned visible light and near-infrared images of each individual, which will be used as the training set for subsequent network training; wherein, the bimodal palm image of each individual includes a visible light palm print image and a near-infrared palm vein image. 2) The training set is fed into the dual-stream deep learning recognition network for iterative training. During training, the visible light image and near-infrared image are first input into the symmetric dual-stream feature extraction path for deep semantic modeling, outputting corresponding modal feature maps at each level. Subsequently, the obtained mid-level modal feature maps are fed into the SDRM for frequency domain discrimination and purification, outputting mid-level refined feature maps. At the same time, the obtained deep modal feature maps are fed into the SK-Fusion for dynamic weight fusion, generating deep fused feature maps. Next, the mid-level refined feature maps and the deep fused feature maps are fed into the pyramid feature fusion architecture for spatial alignment and channel concatenation, constructing multi-scale fused feature maps, which are then input into the DSPM for spatial topology extraction and spatial channel calibration, reducing dimensionality and compressing them into fixed-dimensional discriminative feature vectors. Finally, the joint loss function constraint architecture is used to calculate the joint loss value between the prediction result of the dual-stream deep learning recognition network and the real label. Based on the joint loss value, the parameters of the dual-stream deep learning recognition network are iteratively optimized using the backpropagation algorithm to obtain the trained dual-stream deep learning recognition network. 3) Input the bimodal palm image of the individual to be identified into the trained dual-stream deep learning recognition network. The network automatically extracts the discriminative feature vector of the palm and outputs the identity recognition result of the palm through feature comparison, thus completing the bimodal palm recognition task.
2. The dual-modal palm recognition method based on a dual-stream deep learning recognition network according to claim 1, characterized in that, In step 2), the symmetric dual-stream feature extraction pathway uses EfficientNet-B0 based on the moving-flipping bottleneck convolution MBConv as the basic unit. It learns the surface texture of visible light images and the subcutaneous vein structure of near-infrared images through parameter-independent dual-stream branches. The mid-level modal feature map is extracted from the fourth stage of EfficientNet-B0 and is divided into visible light mid-level features and near-infrared mid-level features according to feature type. Taking advantage of the fact that this stage is at the receptive field expansion node, it preserves local spatial texture and extracts preliminary semantic features. The deep modal feature map is extracted from the end of EfficientNet-B0 and is used to capture highly abstract global identity representation. Together with the mid-level modal feature map, it constitutes a multi-level feature system covering details, structure, and semantics.
3. The dual-modal palm recognition method based on a dual-stream deep learning recognition network according to claim 2, characterized in that, In step 2), the specific processing steps of the SDRM are as follows: Input mid-level modal feature map Mapping to a high-dimensional space, where R represents the real number field, and C, H, and W represent the number of channels, image height, and image width, respectively, yields the feature map. ; Utilizing depthwise separable convolutions on feature maps After performing spatial local modeling, it is split into two parallel branches along the channel dimension. and Using the GELU nonlinear activation function to Process and with Element-wise multiplication yields the gated enhanced feature X. gate Then, linear projection is used to restore the number of channels, thus obtaining the preprocessed features. ; Using rearrangement operators to preprocess features Divided into several sizes Local feature block X patch For each local feature block X patch Perform a real-number fast Fourier transform (rFFT2) to map the frequency from the spatial domain to the frequency domain, obtaining the spectral components containing complex information. ; Construct a learnable complex weight matrix with the same dimension as the spectral components as the frequency discrimination matrix. ;Will and Performing pointwise Hadamard product operations enables adaptive gain control of different frequency components in the spectrum. This adaptive gain control is achieved through... Automatic iterative updates preserve the mid-frequency response that characterizes the palm print ridge features and suppress low-frequency illumination components and high-frequency noise components. Using Inverse Fast Fourier Transform (irFFT2) to extract spectral components Reverting to the spatial domain yields local feature blocks with improved discrimination purity. Finally, the rearrangement operator is used to handle all local feature blocks. Reorganize according to the original spatial location to output the final mid-level refined feature map.
4. The dual-modal palm recognition method based on a dual-stream deep learning recognition network according to claim 3, characterized in that, In step 2), the specific processing steps of SK-Fusion are as follows: The convolution operator is used to adjust the channels of the deep modal feature maps of visible light and near infrared light, and then element-wise addition is performed to fuse them to obtain global descriptive features with preliminary dual-modal interaction information. The visible light and near-infrared deep modal feature maps are simultaneously fed into two sets of deep convolutional branches with different receptive fields: Small kernel branch: Employs 3×3 depthwise convolution to extract fine local feature maps; Large kernel branch: Following the previous convolutional layer, a 5×5 dilated convolution is used to further expand the receptive field and extract a wider range of spatial context semantics; Global average pooling (GAP) is performed on the global descriptive features to obtain channel statistics. Then, a bottleneck layer containing two fully connected layers is used for nonlinear mapping to extract higher-order modality association weight vectors. Specifically, the features of two sets of deep convolutional branches are concatenated using a connection operator, and two dimensions of statistics are extracted simultaneously: first, average pooling and max pooling are performed on the concatenated features to generate a dual-channel spatial descriptor agg; second, global average pooling (GAP) is performed on the concatenated features, and then the features are compressed to the latent space dimension d through a bottleneck structure consisting of 1×1 convolutions, normalization layers, and ReLU activation functions. , It is the vector dimension before compression. It's the compression ratio. It sets a minimum dimension threshold; A 1×1 convolutional layer is used to increase the dimensionality of the latent space features, and Softmax normalization is performed to generate mutually exclusive and complementary adaptive weight coefficients in the modality and kernel dimensions. and The weighting coefficient , The reshape operation is combined element-wise with the spatial descriptor agg to generate the final dynamic selection operator that acts on each branch. and ; Using the dynamic selection operator and The original features of the two sets of deep convolutional branches are weighted and summed pixel by pixel to obtain the fused intermediate features. Finally, the enhanced deep fused feature map is output through a 1×1 convolutional layer and a Sigmoid activation function.
5. The dual-modal palm recognition method based on a dual-stream deep learning recognition network according to claim 4, characterized in that, In step 2), the specific processing steps of the pyramid feature fusion architecture are as follows: Spatially downsample the mid-level refined feature map after SDRM processing using a convolutional layer with a stride of 2, so that its spatial resolution is forcibly aligned with the deep fusion feature map output by SK-Fusion. By using a concatenation operator, the aligned visible light mid-layer features, near-infrared mid-layer features, and deep fused feature maps are cascaded and stacked along the channel dimension to construct a multi-scale fused feature map. .
6. The dual-modal palm recognition method based on a dual-stream deep learning recognition network according to claim 5, characterized in that, In step 2), the specific processing steps of the DSPM are as follows: Input multi-scale fused feature map The input is synchronously fed into two parallel branches that compute features differently, in order to achieve the collaborative extraction of local details and macroscopic topology. The specific details of these parallel branches are as follows: Standard Convolutional Flow: Employs two cascaded 3×3 convolutional layers with a dilation rate of 1 to extract fine yet continuous skin texture and ridge features from the multi-scale fused feature map, outputting a local feature map. ; Dilated Convolutional Flow: Employs two asymmetric dilated convolutional layers. The dilation rate of the first dilated convolutional layer is set to 2, and that of the second dilated convolutional layer is set to 3. By increasing the sampling stride, a larger receptive field is obtained to capture the cross-regional geometric topology of the palm and the relative spatial positions between feature points, outputting a topological feature map. ; Using the join operator and Channel cascading is performed to obtain the cascaded feature map. And using 1×1 convolutional layers to Channel compression is performed to achieve initial intermodal interaction, and the compressed data is then processed. The input undergoes a bottleneck structure consisting of two 1×1 convolutional layers, which is then nonlinearly mapped to generate channel weight vectors. These vectors are then processed sequentially using a channel attention module (CA) and a spatial attention module (SA) to generate feature maps. ; In addition, a perceptual enhancement path parallel to the aforementioned standard convolutional flow and dilated convolutional flow is introduced, along with predefined hyperparameters. As the gain coefficient of the perception enhancement path, it is calculated using the formula... The features in the formula are integrated, where The input features are aligned in dimension after a 1×1 convolution. This represents the enhanced features after processing by the collaborative attention mechanism; To control the balance factor of the enhanced feature ratio, which is used to dynamically adjust the linear fusion ratio of the feature streams of the perception enhancement path, the final fixed-dimensional discriminative feature vector is obtained by performing ReLU nonlinear activation. .
7. The dual-modal palm recognition method based on a dual-stream deep learning recognition network according to claim 6, characterized in that, In step 2), the total loss of the joint loss function constructed by the joint loss function constraint architecture is... The calculation formula is: ,in, To determine the loss based on the angle; To compare similarity loss, For manifold consistency loss, , This is the preset balance coefficient; The angle discrimination loss By introducing an additive angle cosine margin at the angle between the feature vector and the weight vector, the features are mapped to a hyperspherical space to construct a strict angle decision boundary, thereby compressing the intra-class distance and expanding the inter-class distance. The comparison similarity loss By constructing cross-modal positive and negative sample pairs, the following can be achieved: using associated labels to narrow the feature mapping distance of the same identity in visible light images and near-infrared images, while widening the modal feature distance between individuals with different identities, thus explicitly eliminating the identity representation gap between modalities; The manifold consistency loss The mean square error operator is used to constrain the visible light and near-infrared modal features of the same individual to maintain a high degree of overlap in the manifold space, ensuring that the network has manifold stability against the unique noise under different modes.
8. The dual-modal palm recognition method based on a dual-stream deep learning recognition network according to claim 7, characterized in that, The balance coefficient , The balance coefficient is heuristically adjusted based on the magnitude of the loss during the training phase. The value range is 0.1 to 0.
5. The value range is 0.01 to 0.1.