A multi-stage feature fusion-based cross-modal identity authentication method

Abstract

The present application relates to the technical field of identity authentication, in particular to a cross-modal identity authentication method based on multi-stage feature fusion, which adopts a multi-stage feature fusion network MSFFNet, the MSFFNet adopts a text-image double-flow structure network, so that two modalities learn bottom features respectively; three adjacent layer feature fusion modules AFFM are embedded between Layer1 and Layer3, each AFFM module internally comprises a multi-scale shallow layer extraction unit MSEU, adaptive guidance of deep layer semantics to shallow layer details is realized, so that the fusion features have both detail and semantic expression capabilities; an attention enhancement module is introduced, global learnable prior knowledge is used to optimize features and focus on user saliency areas, and background interference is suppressed. Through the above progressive interaction and focusing mechanism, the text-image modal difference is effectively bridged, and the problems of insufficient single level feature expression capability and difficulty in effective interaction between shallow layer details and deep layer semantics are solved.

Description

Technical Field

[0001] This invention relates to the field of authentication technology, and more specifically to a cross-modal authentication method based on multi-stage feature fusion. Background Technology

[0002] In scenarios such as the Internet of Things (IoT), mobile payments, and remote work, continuous user authentication is a core element in ensuring system security. Traditional single-modal authentication methods (such as text-only passwords or facial recognition-only) are susceptible to environmental interference or forgery attacks. Multimodal authentication improves security and robustness by fusing different types of information (such as passwords + facial recognition). However, significant heterogeneity exists between different modalities (text and images): text is discrete, high-level semantic symbols, while images are continuous, low-level pixel matrices. This fundamental difference leads to severe inconsistencies in the feature distribution of the same user across different modalities.

[0003] To address these challenges, existing technologies propose feature fusion-based methods to reduce intermodal differences by aligning the feature spaces of different modalities. For example, feature-level concatenation is the most direct method, directly concatenating feature vectors extracted from different modalities along the channel dimension to form a joint feature vector, which is then input into a classifier for identity determination. Contrastive learning-based alignment uses a self-supervised learning mechanism that brings positive sample pairs closer and pushes negative sample pairs further apart, mapping features from different modalities to a unified semantic space, achieving coarse-grained alignment between modalities. Attention-based feature fusion calculates element-level association weights between modalities, dynamically focusing on semantically relevant local regions to achieve fine-grained feature interaction and fusion. However, these methods rely on the quality of feature extraction; single-level features often struggle to simultaneously capture both detailed and semantic information, limiting further improvements in recognition performance. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a cross-modal authentication method based on multi-stage feature fusion, which solves the problems of insufficient single-level feature expression capability and difficulty in effective interaction between shallow details and deep semantics.

[0005] The technical solution adopted by this invention to solve its technical problem is: A cross-modal authentication method based on multi-stage feature fusion includes: Design a multi-stage feature fusion network MSFFNet, which adopts a text-image dual-stream structure; Text Stream: The input text sequence first passes through a text encoder to obtain a global text feature vector. The global text feature vector is first mapped to the same channel dimension as the image features through a linear projection layer to obtain the projected text features. Then, the projected text features are expanded in dimension and expanded along the spatial dimension using a broadcast copy operation to generate a text feature map, so that the text features and image features have the same tensor dimension, which meets the requirements of subsequent cross-modal attention interaction. Image stream: Image input uses ResNet-50 pre-trained on ImageNet as the backbone; to address the modal differences between text and images, the network adopts a non-parameter-sharing strategy in Layer 0 of ResNet-50 to extract shallow features of the two modalities separately; in Layers 1 to 3, a parameter-sharing strategy is adopted to learn the deep features common to the two modalities. Three adjacent feature fusion modules (AFFM) are embedded between Layer 1 and Layer 3: AFFM-1, AFFM-2, and AFFM-3. This achieves precise cross-modal semantic alignment by aggregating features from different layers. Specifically: AFFM-1: The text feature map is used as the query Q, and the image feature map output by ResNet-50 Layer 1 is used as the key K and value V; AFFM-2: The output of AFFM-1 is used as the query Q, and the image feature map output by ResNet-50 Layer 2 is used as the key K and value V; AFFM-3: The output of AFFM-2 is used as the query Q, and the image feature map output by ResNet-50 Layer 3 is used as the key K and value V; Since the spatial resolution of the output feature maps of ResNet-50 differs at different stages, in order to ensure the effectiveness of cross-stage feature interaction, the spatial size of the fused feature output by each AFFM module is consistent with the stage deep feature after linear projection of the current global text feature vector; specifically, the output size of AFFM-1 is consistent with the size of the Layer2 feature map, the output size of AFFM-2 is consistent with the size of the Layer3 feature map, and the output size of AFFM-3 is consistent with the size of the Layer3 feature map, thereby ensuring dimensionality matching in the subsequent fusion process; Finally, the Attention Enhancement Module (AEM) is applied to optimize the fused features and focus on key regions, highlighting user salient features, suppressing background interference, and outputting the authentication results through a classifier.

[0006] In the above technical solution, a multi-stage feature fusion network is used to achieve progressive interactive fusion of text semantics and image details. An attention enhancement module is used to optimize the fused features and focus on key regions. While preserving fine-grained discriminative information, the robustness and discriminativeness of the features are improved, which solves the problems of insufficient expression ability of single-level features and difficulty in effective interaction between shallow details and deep semantics.

[0007] Furthermore, the neighboring layer feature fusion module is used to aggregate features from different levels, fully utilizing shallow image detail information and deep / textual semantic information for cross-modal interaction, thereby enhancing the ability to identify user identities. The structure of the neighboring layer feature fusion module is as follows: shallow image features F l ∈ R H’×W’×C’ and deep fusion features F h ∈ R H×W×C As input, where C The number of channels representing the feature. H and W These represent height and width, respectively. We designed a Multi-scale Shallow Extraction Unit (MSEU) and introduced a convolutional layer to extract shallow image features through the MSEU. F l Perform multi-scale feature extraction: First, shallow image features F l Two parallel convolutional paths are used respectively: F 1 l = Conv 1×1 (Conv 5×5 ( F l )); F 2 l = Conv 1×1 (Conv 7×7 ( F l )); Among them, Conv 5×5 and Conv 7×7 These represent convolution operations with kernel sizes of 5 and 7, respectively, used to capture spatial pixel relationships over a wider range, thereby more effectively extracting the user's local texture patterns and structural details. Subsequently, a 1×1 convolution is used to perform channel compression and linear transformation on the extracted features. Larger convolution kernels can expand the receptive field, making up for the shortcomings of small convolution kernels in modeling long-distance pixel dependencies, so that subtle but crucial discriminative information in shallow features can be more fully preserved. Next, the features output by the two paths are... F 1 l and F 2 l The result is obtained by splicing along the channel dimension. F 12 l ∈ R H’×W’×2C’ : F 12 l = Cat( F 1 l , F 2 l ); Multi-scale features after splicing F 12 l Perform global average pooling and global max pooling respectively to aggregate the spatial dimensionality information of the feature map: F avg = Avg ( F 12 l ); F max = Max ( F 12 l ); in, Avg and Max These represent global max pooling and global average pooling performed along the channel direction, respectively, resulting in two shapes. H ’ × W ’ A feature map of ×1. Average pooling preserves the overall response of the feature, while max pooling captures the most salient feature response. These two feature maps are then compared with the original multi-scale features. F2 l and F 1 l Element-wise multiplication is performed to enhance important channels and suppress redundant channels: F 2’ l = F avg ☉ F 2 l ; F 1’ l = F max ☉ F 1 l ; Here, ☉ represents element-wise multiplication. In this way, features from different receptive fields each acquire corresponding channel attention weights, allowing the model to dynamically adjust the feature contribution based on different input features. Subsequently, the enhanced features from the two paths are added and fused, then integrated through a 1×1 convolution, and finally combined with the original shallow features. F l Residual connections yield enhanced shallow features. F ’ l ∈ R H’×W’×C’ : F ’ l = Conv 1×1 ( F 1’ l + F 2’ l )+ F l ; Introducing an average pooling layer for size alignment: F pool l = AvgPool ( F ’ l ); in, AvgPool Using average pooling with a kernel size of 2 and a stride of 2, we obtain enhanced shallow features with a size equal to that of the deep feature space. F pool l ∈R H×W×C’ This is used for subsequent feature fusion.

[0008] Furthermore, after completing the multi-scale extraction and channel enhancement of shallow features, an attention mechanism is introduced to perform pixel-level fusion of shallow and deep features: First, the input features are projected using two independent 1×1 convolutional layers and an MSEU module to achieve deep feature fusion. F h The query tensor is obtained using a convolutional layer with a kernel size of 1×1. F q ∈ R H×W×C’ ,right F l The bond tensor is obtained by using a 1×1 convolution. F k ∈ R H’×W’×C’ And using the MSEU module to obtain value tensors F v ∈ R H’×W’×C’ : F q = Conv 1×1 ( F h ); F k = Conv 1×1 ( F l ); F v =MSEU( F l ); To achieve pixel-level interaction between shallow and deep features, the query tensor... F q Reshape to size HW × C The matrix of ', the key tensor F k Reshape to size H ' W '× C The matrix ', then the query tensor F q transpose of the key tensor F T k ∈ R C’×H’W’Perform matrix multiplication and normalize using the Softmax function to obtain the attention weights. A ∈ R HW×H’W’ : ; in, Represents matrix multiplication. This is a scaling factor used to stabilize gradient calculation. The attention weight A reflects the correlation between each location in the deep feature and all locations in the shallow feature; Using attention weights on the value tensor of shallow features F v Weighted aggregation is performed to obtain the interaction feature representation between deep and shallow layers. F agg ∈ R HW×C’ : ; Finally, the deep and shallow interaction features F agg Remodeling H × W × C ’ Then, combined with enhanced shallow features F pool l Element-wise addition is performed, and the channel number is restored to C using a 1×1 convolution to obtain the features of the adjacent layers. F : F = Conv 1×1 ( F agg + F pool l ); It achieves effective interaction between shallow detailed information and deep semantic information.

[0009] Furthermore, the structure of the attention enhancement module is as follows: The neighboring layer features F ∈ R C×H×W Remodeling F A ∈ R N×C , N = H × W Indicates the total number of pixels. C This refers to the number of channels in the feature map. Two learnable storage units are established, which are implemented using a fully connected layer and denoted as external key matrices, respectively. Mk ∈ R S×C and external value matrix M v ∈ R S×C , S This represents the number of nodes in the fully connected layer. Will F A With external key matrix M k The transpose of the matrix is ​​used to perform matrix multiplication to generate the original attention map. E ∈ R N×S : ; in, Representing matrix multiplication, attention graph E No. N Line number S The column element represents the first N Pixels and external bond matrix M k No. S The degree of correlation of row-learnable features enables the association and interaction between all pixels and learnable storage units; For the original attention map E Perform Softmax normalization on each row to obtain a row-normalized attention map. a ∈ R N×S : ; in, i Corresponding to different pixel positions, j The denominator represents the different learnable units corresponding to the external bond matrix. i The sum of the original association values ​​of all learnable units in each row; this normalization operation makes the sum of all elements in each row equal to 1, highlighting the first... i The relative correlation strength between learnable units in different rows of the pixel and the external key matrix enhances the robustness of the attention mechanism; Then, the attention map after row normalization. a Each column is L1 normalized to obtain the final attention map. A ∈ R N×S : ; in, i Corresponding to different pixel positions, j The denominator represents the different learnable units corresponding to the external bond matrix. jThe sum of attention values ​​for all pixels in each column; this normalization operation makes the sum of all elements in each column equal to 1, thus highlighting the first pixel. j The relative correlation strength between each learnable storage unit and all pixels; through this bidirectional normalization operation, the model can effectively balance the response strength between different pixels, playing a role in strong regularization; Finally, the normalized attention map is used. A With external value matrix M v Perform weighted aggregation to combine feature maps F Updated to reshape features F A ∈ R N×C : ; By introducing an external attention module, the network can use prior knowledge learned from the entire dataset to guide the feature refinement of the current sample, enabling it to accurately align and enhance the target semantic information while suppressing background noise.

[0010] Furthermore, the MSFFNet network is jointly optimized using cross-entropy loss and hard-sample triplet loss: By aggregating samples from different modalities of the same user in the feature space, cross-entropy loss can effectively enhance the model's robustness to intra-class discrepancies. The formula for calculating cross-entropy loss is as follows: ; in, N y represents the number of pedestrian identities, and y represents the true class label of the input sample. p i Is the input sample the first? i Predicted probability for each user h i Indicates sample i Is the predicted value the true label? If it is the true label, then... h i =1; otherwise h i = 0; The hard sample triplet loss is used to identify the most difficult-to-distinguish positive and negative samples in a batch of data. The formula for calculating the hard sample triplet loss is as follows: ; in, P Indicates the number of user identities. K This represents the number of samples for each user. α The boundary parameters represent the positive and negative samples. a i , pi , n j These represent anchor points, positive samples, and negative samples, respectively. The total loss function is: .

[0011] Technical effects of the present invention: Compared with existing technologies, the cross-modal authentication method based on multi-stage feature fusion of the present invention has the following advantages: 1. Significantly improves the expressive power of cross-modal features: By designing the multi-stage feature fusion network MSFFNet, three neighboring layer feature fusion modules AFFM are embedded between Layer 1 and Layer 3 of ResNet-50, realizing the progressive interaction between text semantics and image details. This overcomes the problem that single-level features cannot simultaneously take into account both detailed information and semantic information, and significantly improves the ability to distinguish user identity. 2. Effective integration of shallow details and deep semantic information: The AFFM module is designed with a multi-scale shallow extraction unit (MSEU), which extracts multi-scale spatial features through large convolutional kernels (5×5, 7×7) and enhances fine-grained discriminative information by combining channel attention mechanism; at the same time, a pixel-level attention mechanism is introduced, which enables deep semantic features to adaptively guide the integration of shallow detail features, thus solving the technical problem that shallow details and deep semantics are difficult to interact effectively. 3. Enhance the robustness and anti-interference ability of features: Introduce the attention enhancement module AEM, which utilizes globally learnable external key-value storage units to perform bidirectional normalized attention optimization on fused features under the guidance of prior knowledge of the entire dataset. This effectively suppresses redundant information caused by background noise and modality differences, accurately focuses on user salient regions, and improves the robustness of features in complex environments. 4. Optimize the discrimination performance of cross-modal authentication: Adopt a joint optimization strategy of cross-entropy loss and hard sample triplet loss to reduce the feature distance between text and image samples of the same user while increasing the feature distance between samples of different users, thereby further reducing the differences between modalities and improving the inter-class discrimination, thus obtaining better authentication accuracy. Attached Figure Description

[0012] Figure 1 This is a flowchart of the cross-modal authentication method based on multi-stage feature fusion of the present invention; Figure 2 This is a structural diagram of the adjacent layer feature fusion module of the present invention; Figure 3 This is a structural diagram of the attention enhancement module of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0014] Example 1: like Figure 1 As shown, this embodiment relates to a cross-modal authentication method based on multi-stage feature fusion, including: Design a multi-stage feature fusion network MSFFNet, which adopts a text-image dual-stream structure; Text stream: The input text sequence (such as a user password or identity description) is first processed by a text encoder to obtain a global text feature vector T∈R. D The global text feature vector T is first mapped to the same channel dimension C as the image features through a linear projection layer, resulting in the projected text features T. p ∈R C Then T p Dimensional expansion is performed, and a broadcast copy operation is used to expand along the spatial dimensions to generate text feature maps: F text ∈R C×H×W ; Here, H and W correspond to the spatial dimensions of the image feature maps received by AFFM-1, thus ensuring that the text features and image features have the same tensor dimension, which meets the requirements of subsequent cross-modal attention interaction.

[0015] Image Stream: Image input uses ResNet-50 pre-trained on ImageNet as its backbone. To address the modal differences between text and images, the network employs a non-parameter-sharing strategy in Layer 0 of ResNet-50 to extract shallow features specific to each modality. In Layers 1 to 3, a parameter-sharing strategy is used to learn deep features common to both modalities. It should be noted that the network described in this invention only uses the first three stages of ResNet-50 (Layer 0, Layer 1, Layer 2, and Layer 3). Layer 4 (outputting a 7×7 feature map) is not included in the fusion branch because its spatial resolution is too low to effectively perform pixel-level cross-modal attention interaction.

[0016] Three adjacent feature fusion modules (AFFM) are embedded between Layer 1 and Layer 3: AFFM-1, AFFM-2, and AFFM-3. This achieves precise cross-modal semantic alignment by aggregating features from different layers. Specifically: AFFM-1: Uses text feature maps as query Q and image feature maps output by ResNet-50Layer1 as key K and value V; AFFM-2: The output of AFFM-1 is used as the query Q, and the image feature map output by ResNet-50 Layer 2 is used as the key K and value V; AFFM-3: The output of AFFM-2 is used as the query Q, and the image feature map output by ResNet-50 Layer 3 is used as the key K and value V; Since the spatial resolution of the output feature maps of ResNet-50 differs at different stages, in order to ensure the effectiveness of cross-stage feature interaction, the spatial size of the fused feature output by each AFFM module is consistent with the stage deep feature after linear projection of the current global text feature vector; specifically, the output size of AFFM-1 is consistent with the size of the Layer2 feature map, the output size of AFFM-2 is consistent with the size of the Layer3 feature map, and the output size of AFFM-3 is consistent with the size of the Layer3 feature map, thereby ensuring dimensionality matching in the subsequent fusion process; Finally, the Attention Enhancement Module (AEM) is applied to optimize the fused features and focus on key regions, highlighting user salient features, suppressing background interference, and outputting the authentication results through a classifier.

[0017] 1. Neighboring layer feature fusion module To address the issue of insufficient feature representation capability at a single level, this invention designs an adjacent layer feature fusion module (AFFM), the structure of which is as follows: Figure 2 As shown, this module aims to aggregate features from different levels, fully utilizing shallow image details and deep / textual semantic information for cross-modal interaction, thereby enhancing the ability to identify user identities. Furthermore, considering that shallow image feature maps contain more pixel-level details about user contours, textures, and edges—details crucial for distinguishing users with different identities—the module design specifically strengthens its ability to extract information from shallow features.

[0018] AFFM uses shallow image features F l ∈ R H’×W’×C’ and deep fusion features F h ∈ R H×W×C As input, where C The number of channels representing the feature. H and WThese represent height and width, respectively. Shallow image features typically contain rich details such as texture and edges, while deep fusion features encode higher-level semantic information, such as identity attributes in text descriptions or high-level representations after fusion in the previous stage. Effective fusion of the two helps improve the discriminative ability of cross-modal authentication.

[0019] To extract richer spatial structure information from shallow features, this invention designs a multi-scale shallow extraction unit (MSEU). For example... Figure 2 As shown, for shallow image features containing rich pixel-level information F l MSEU introduces convolutional layers with large kernels for multi-scale feature extraction. Specifically, it extracts shallow image features. F l First, two parallel convolutional paths are used: F 1 l = Conv 1×1 (Conv 5×5 ( F l )); F 2 l = Conv 1×1 (Conv 7×7 ( F l )); Among them, Conv 5×5 and Conv 7×7 The numbers represent convolution operations with kernel sizes of 5 and 7, respectively, used to capture spatial pixel relationships over a wider range, thereby more effectively extracting the user's local texture patterns and structural details. Subsequently, a 1×1 convolution is used to perform channel compression and linear transformation on the extracted features. Larger kernels can expand the receptive field, compensating for the shortcomings of smaller kernels in modeling long-distance pixel dependencies, allowing subtle but crucial discriminative information in shallow features to be more fully preserved.

[0020] Next, the features output by the two paths are... F 1 l and F 2 l The result is obtained by splicing along the channel dimension. F 12 l ∈ R H’×W’×2C’ : F 12 l = Cat( F 1 l , F 2 l ); To further enhance the channel discriminative power of shallow features and make the model pay more attention to channel information important for cross-modal authentication tasks, this invention introduces channel attention. Specifically, for the concatenated multi-scale features... F 12 l Perform global average pooling and global max pooling respectively to aggregate the spatial dimensionality information of the feature map: F avg = Avg ( F 12 l ); F max = Max ( F 12 l ); in, Avg and Max These represent global max pooling and global average pooling performed along the channel direction, respectively, resulting in two shapes. H ’ × W ’ A feature map of ×1. Average pooling preserves the overall response of the feature, while max pooling captures the most salient feature response. These two feature maps are then compared with the original multi-scale features. F 2 l and F 1 l Element-wise multiplication is performed to enhance important channels and suppress redundant channels: F 2’ l = F avg ☉ F 2 l ; F 1’ l = F max ☉ F 1 l ; Here, ☉ represents element-wise multiplication. In this way, features from different receptive fields each acquire corresponding channel attention weights, allowing the model to dynamically adjust the feature contribution based on different input features. Subsequently, the enhanced features from the two paths are added and fused, then integrated through a 1×1 convolution, and finally combined with the original shallow features. F l Residual connections yield enhanced shallow characterization. F ’ l ∈ R H’×W’×C’ : F ’ l = Conv 1×1 ( F 1’ l + F 2’ l )+ F l ; To achieve subsequent feature fusion, it is necessary to... F ’ l Spatial dimensions downsampled to F h Consistent. Therefore, an average pooling layer is introduced after the residual connection for size alignment: F pool l = AvgPool ( F ’ l ); in AvgPool Using average pooling with a kernel size of 2 and a stride of 2, we obtain enhanced shallow features with a size equal to that of the deep feature space. F pool l ∈ R H×W×C’ This is used for subsequent feature fusion.

[0021] After completing multi-scale extraction and channel enhancement of shallow features, this invention introduces an attention mechanism to perform pixel-level fusion of shallow and deep semantic information to further achieve effective interaction between shallow detail information and deep semantic information. First, the input features are projected using two independent 1×1 convolutional layers and an MSEU module. F h The query tensor is obtained using a convolutional layer with a kernel size of 1×1. F q∈ R H×W×C’ ,right F l The bond tensor is obtained by using a 1×1 convolution. F k ∈ R H’×W’×C’ And using the MSEU module to obtain value tensors F v ∈ R H’×W’×C’ : F q = Conv 1×1 ( F h ); F k = Conv 1×1 ( F l ); F v =MSEU( F l ); To achieve pixel-level interaction between shallow and deep features, the query tensor... F q Reshape to size HW × C The matrix of ', the key tensor F k Reshape to size H ' W '× C The matrix ', then the query tensor F q transpose of the key tensor F T k ∈ R C’×H’W’ Perform matrix multiplication and normalize using the Softmax function to obtain the attention weights. A ∈ R HW×H’W’ : ; in, Represents matrix multiplication. `A` is a scaling factor used to stabilize gradient calculation. The attention weights `A` reflect the correlation between each location in the deep features and all locations in the shallow features. The attention weights are used to construct the value tensor of the shallow features. F v Weighted aggregation is performed to obtain the interaction feature representation between deep and shallow layers. Fagg ∈ R HW×C’ : ; Finally, the deep and shallow interaction features F agg Remodeling H × W × C ’ Then, it is combined with the shallow features after multi-scale extraction and channel weighting. F pool l Element-wise summation is performed, and then a 1×1 convolution is used to restore the number of channels to C, yielding the features of the adjacent layers. F : F = Conv 1×1 ( F agg + F pool l ).

[0022] 2. Attention Enhancement Module After fusing features from neighboring layers using the AFFM module, this invention introduces AEM into the network to further refine the fused features and remove redundant information introduced by modal differences and background interference. This module captures the spatial correlation between pixels by introducing two shared learnable units, thereby achieving accurate localization of target features.

[0023] The structure diagram of the attention enhancement module is as follows: Figure 3 As shown, the neighboring layer features obtained in the previous stage are... F ∈ R C×H×W Remodeling F A ∈ R N×C , N = H × W Indicates the total number of pixels. C Let be the number of channels in the feature map. The module establishes two learnable storage units, implemented using fully connected layers, denoted as the outer key matrix. M k ∈ R S×C and external value matrix M v ∈ R S×C , S This represents the number of nodes in the fully connected layer.

[0024] To achieve the association and interaction between all pixels and learnable storage units, F A With external key matrix M k The transpose of the matrix is ​​used to perform matrix multiplication to generate the original attention map. E ∈ R N×S : ; in, Representing matrix multiplication, attention graph E No. N Line number S The column element represents the first N Pixels and external bond matrix M k No. S The degree of correlation between learnable features.

[0025] To enhance the robustness of the attention mechanism, the original attention map was modified. E Perform Softmax normalization on each row to obtain a row-normalized attention map. a ∈ R N×S : ; in, i Corresponding to different pixel positions, j The denominator represents the different learnable units corresponding to the external bond matrix. i The sum of the original association values ​​of all learnable units in each row. This normalization operation makes the sum of all elements in each row equal to 1, highlighting the first... i The relative correlation strength between learnable units in each pixel that are not in the same row as the external key matrix.

[0026] Then, the attention map after row normalization. a Each column is L1 normalized to obtain the final attention map. A ∈ R N×S : ; in, i Corresponding to different pixel positions, j The denominator represents the different learnable units corresponding to the external bond matrix. j The sum of attention values ​​for all pixels in each column. This normalization operation makes the sum of all elements in each column equal to 1, thus highlighting the attention value of the first pixel. j The relative correlation strength between each learnable storage unit and all pixels.

[0027] Through this bidirectional normalization operation, the model can effectively balance the response intensity between different pixels, achieving a strong regularization effect. Finally, the normalized attention map is used... A With external value matrix M v Perform weighted aggregation to combine feature maps F Updated to refined remodeling features F A ∈ R N×C : ; By introducing an external attention module, the network can leverage prior knowledge learned across the entire dataset to guide feature refinement for the current sample. In cross-modal identity verification tasks, this mechanism helps identify salient contextual features shared across different modalities, achieving accurate alignment and enhancement of pedestrian target semantic information while suppressing background noise.

[0028] 3. Loss Function Loss functions are crucial in cross-modal authentication. They guide network parameter optimization by quantifying the difference between model predictions and ground truth labels, enabling the model to effectively narrow the feature distance between text-image samples from the same user while widening the feature distance between samples from different users. Cross-entropy loss and triplet loss are two of the most commonly used loss functions in this task. To further reduce cross-modal differences and improve inter-class discriminability, this invention employs cross-entropy loss and hard-sample triplet loss to jointly optimize the MSFFNet network.

[0029] (1) Cross-entropy loss Cross-entropy loss, also known as identity loss, is a loss function that measures the deviation between the model's predicted output and the true label, making it particularly suitable for classification tasks. By aggregating samples from different modalities of the same user in the feature space, cross-entropy loss effectively enhances the model's robustness to intra-class discrepancies. The formula for calculating cross-entropy loss is as follows: ; in, N y represents the number of pedestrian identities, and y represents the true class label of the input sample. p i Is the input sample the first? i Predicted probability for each user h i Indicates sample i Is the predicted value the true label? If it is the true label, then... h i =1; otherwise h i = 0.

[0030] (2) Loss on hard sample triples Hard sample triplet loss is a commonly used loss function in cross-modal identity verification tasks. Its core lies in identifying the most difficult-to-distinguish positive and negative samples in a batch of data. For each anchor point, this loss function selects the farthest positive sample of the same identity and the closest negative sample of a different identity. By optimizing the relative distance between these three, the model is forced to focus on sample pairs that are difficult to identify correctly, thereby making the distance between positive samples of the same identity closer and the distance between negative samples of different identities farther.

[0031] ; in, P Indicates the number of user identities. K This represents the number of samples for each user. α The boundary parameters represent the positive and negative samples. a i , p i , n j These represent anchor points, positive samples, and negative samples, respectively.

[0032] In summary, the total loss function is: .

[0033] This invention is based on MSFFNet using ResNet-50. Addressing the modal differences between text and image, the network employs a Transformer encoder to extract semantic features for the text modality, while using a non-shared parameter structure in Layer 0 of ResNet-50 for the image modality. This allows each modality to learn low-level features separately, preserving their respective detailed information. Three neighboring feature fusion modules (AFFM) are embedded between Layer 1 and Layer 3. Each AFFM module contains a multi-scale shallow extraction unit (MSEU), which extracts multi-scale spatial information using large convolutional kernels and introduces channel attention to enhance fine-grained features. Simultaneously, the attention mechanism adaptively guides deep semantics to shallow details, ensuring the fused features possess both detailed and semantic expressive capabilities. Subsequently, an attention enhancement module (AEM) is introduced, utilizing globally learnable prior knowledge to optimize features and focus on user-saliency regions, suppressing background interference. The loss function employs a joint optimization of cross-entropy loss and cross-modal triplet loss, consolidating identity classification while narrowing the distance between text-image samples of the same user and widening the distance between different users. Through the aforementioned progressive interaction and focusing mechanism, MSFFNet effectively bridges the text-image modality difference, constructs a cross-modal identity representation space that is both discriminative and robust, and solves the problems of insufficient single-level feature expression capabilities and difficulty in effective interaction between shallow details and deep semantics.

[0034] The above-described specific embodiments are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the above-described specific embodiments. Any appropriate changes or modifications made by a person skilled in the art that conform to the claims of the present invention should fall within the patent protection scope of the present invention.

Claims

1. A cross-modal authentication method based on multi-stage feature fusion, characterized in that, include: Design a multi-stage feature fusion network MSFFNet, which adopts a text-image dual-stream structure; Text Stream: The input text sequence first passes through a text encoder to obtain a global text feature vector; the global text feature vector is first mapped to the same channel dimension as the image features through a linear projection layer to obtain the projected text features; then the projected text features are dimensionally expanded and expanded along the spatial dimension using a broadcast copy operation to generate a text feature map; Image stream: The image input uses ResNet-50 pre-trained on ImageNet as the backbone; a non-parameter-sharing strategy is adopted in Layer 0 of ResNet-50 to extract shallow features of the two modalities separately; a parameter-sharing strategy is adopted in Layer 1 to Layer 3 to learn the deep features common to the two modalities. Three neighboring layer feature fusion modules (AFFM) are embedded between Layer 1 and Layer 3: AFFM-1, AFFM-2, and AFFM-3. Accurate cross-modal semantic alignment is achieved by aggregating features from different layers. Specifically: AFFM-1: The text feature map is used as the query Q, and the image feature map output by ResNet-50 Layer 1 is used as the key K and value V; AFFM-2: The output of AFFM-1 is used as the query Q, and the image feature map output by ResNet-50 Layer 2 is used as the key K and value V; AFFM-3: The output of AFFM-2 is used as the query Q, and the image feature map output by ResNet-50 Layer 3 is used as the key K and value V; Among them, the output size of AFFM-1 is the same as the feature map size of Layer2, the output size of AFFM-2 is the same as the feature map size of Layer3, and the output size of AFFM-3 is the same as the feature map size of Layer3. Finally, the Attention Enhancement Module (AEM) is applied to optimize the fused features and focus on key regions, and the authentication results are output through a classifier.

2. The cross-modal authentication method based on multi-stage feature fusion according to claim 1, characterized in that, The adjacent layer feature fusion module is used to aggregate features from different levels, and its structure is as follows: shallow image features F l ∈ R H’×W’×C’ and deep fusion features F h ∈ R H×W×C As input, where C The number of channels representing the feature. H and W These represent height and width, respectively. We design a multi-scale shallow extraction unit (MSEU) and introduce convolutional layers to extract shallow image features through the MSEU. F l Perform multi-scale feature extraction: First, shallow image features F l Two parallel convolutional paths are used respectively: F 1 l = Conv 1×1 (Conv 5×5 ( F l )); F 2 l = Conv 1×1 (Conv 7×7 ( F l )); wherein, Conv 5×5 and Conv 7×7 represent convolution operations with kernel size 5 and 7, respectively; Next, the features output by the two paths are... F 1 l and F 2 l The result is obtained by splicing along the channel dimension. F 12 l ∈ R H’×W’×2C’ : F 12 l = Cat( F 1 l , F 2 l ); Multi-scale features after splicing F 12 l Perform global average pooling and global max pooling respectively to aggregate the spatial dimensionality information of the feature map: F avg = Avg ( F 12 l )； F max = Max ( F 12 l )； in, Avg and Max These represent global max pooling and global average pooling performed along the channel direction, respectively, resulting in two shapes. H ’ × W ’ A feature map of ×1; Compare these two feature maps with the original multi-scale features respectively. F 2 l and F 1 l Perform element-wise multiplication: F 2’ l = F avg ☉ F 2 l ； F 1’ l = F max ☉ F 1 l ； Where ☉ represents element-wise multiplication; Subsequently, the enhanced features from the two paths are added and fused, and then integrated through a 1×1 convolution, before being combined with the original shallow features. F l Residual connections yield enhanced shallow features. F ’ l ∈ R H’×W’×C’ : F ’ l = Conv 1×1 ( F 1’ l + F 2’ l )+ F l ; Introducing an average pooling layer for size alignment: F pool l = AvgPool ( F ’ l )； in, AvgPool Using average pooling with a kernel size of 2 and a stride of 2, we obtain enhanced shallow features with a size equal to that of the deep feature space. F pool l ∈ R H×W×C’ .

3. The cross-modal authentication method based on multi-stage feature fusion according to claim 2, characterized in that, Introducing an attention mechanism to perform pixel-level fusion of deep and shallow features: First, the input features are projected using two independent 1×1 convolutional layers and an MSEU module to achieve deep feature fusion. F h The query tensor is obtained using a convolutional layer with a kernel size of 1×1. F q ∈ R H×W×C’ ,right F l The bond tensor is obtained by using a 1×1 convolution. F k ∈ R H’×W’×C’ And using the MSEU module to obtain value tensors F v ∈ R H’×W’×C’ : F q = Conv 1×1 ( F h ); F k = Conv 1×1 ( F l ); F v = MSEU( F l ); query tensor F q Reshape to size HW × C The matrix of ', the key tensor F k Reshape to size H ' W '× C The matrix ', then the query tensor F q transpose of the key tensor F T k ∈ R C’×H’W’ Perform matrix multiplication and normalize using the Softmax function to obtain the attention weights. A ∈ R HW×H’W’ : ； in, Represents matrix multiplication. This is the scaling factor; Using attention weights on the value tensor of shallow features F v Weighted aggregation is performed to obtain the interaction feature representation between deep and shallow layers. F agg ∈ R HW×C’ : ； Finally, the deep and shallow interaction features F agg Remodeling H × W × C ’ Then, combined with enhanced shallow features F pool l Element-wise addition is performed, and the channel number is restored to C using a 1×1 convolution to obtain the features of the adjacent layers. F : F = Conv 1×1 ( F agg + F pool l )。 4. The cross-modal authentication method based on multi-stage feature fusion according to claim 3, characterized in that, The structure of the attention enhancement module is as follows: The neighboring layer features F ∈ R C×H×W Remodeling F A ∈ R N×C , N = H × W Indicates the total number of pixels. C This refers to the number of channels in the feature map. Two learnable storage units are established, which are implemented using a fully connected layer and denoted as external key matrices, respectively. M k ∈ R S×C and external value matrix M v ∈ R S×C , S This represents the number of nodes in the fully connected layer. Will F A With external key matrix M k The transpose of the matrix is ​​used to perform matrix multiplication to generate the original attention map. E ∈ R N×S : ； in, Representing matrix multiplication, attention graph E No. N Line number S The column element represents the first N Pixels and external bond matrix M k No. S The degree of correlation between learnable features; For the original attention map E Perform Softmax normalization on each row to obtain a row-normalized attention map. a ∈ R N×S : ； in, i Corresponding to different pixel positions, j The denominator represents the different learnable units corresponding to the external bond matrix. i The sum of the original association values ​​of all learnable units in the row; Attention map after row normalization a Each column is L1 normalized to obtain the final attention map. A ∈ R N×S : ； in, i Corresponding to different pixel positions, j The denominator represents the different learnable units corresponding to the external bond matrix. j The sum of the attention values ​​for all pixels in the column; Using the normalized attention map A With external value matrix M v Perform weighted aggregation to combine feature maps F Updated to reshape features F A ∈ R N×C : 。 5. The cross-modal authentication method based on multi-stage feature fusion according to any one of claims 1-4, characterized in that, The MSFFNet network is optimized using a combination of cross-entropy loss and hard-sample triplet loss. The formula for calculating cross-entropy loss is: ； in, N y represents the number of pedestrian identities, and y represents the true class label of the input sample. p i Is the input sample the first? i Predicted probability for each user h i Indicates sample i Is the predicted value the true label? If it is the true label, then... h i =1; otherwise h i = 0; The formula for calculating the loss of hard sample triples is: ； in, P Indicates the number of user identities. K This represents the number of samples for each user. α The boundary parameters represent the positive and negative samples. a i , p i , n j These represent anchor points, positive samples, and negative samples, respectively.

6. The cross-modal authentication method based on multi-stage feature fusion according to claim 5, characterized in that, The total loss function is: 。

Images