Modal guidance fusion method based on visible light and infrared pedestrian re-identification
By employing the ResNet50 model to extract visible light and infrared video sequence features in pedestrian re-identification, and utilizing a bidirectional modality guidance mechanism and loss function optimization, the modality difference problem was solved, achieving stable and efficient pedestrian identification under different lighting conditions.
Patent Information
- Application Number
- CN202511769379.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-02-17
AI Technical Summary
Existing technologies fail to effectively utilize the complementarity between modes in pedestrian re-identification using visible light and infrared modes, resulting in performance degradation in low light or nighttime environments and high computational requirements. Existing methods have failed to effectively address the problem of modal differences.
We employ a ResNet50 model to extract frame-level spatial features from visible light and infrared video sequences. Infrared feature enhancement is guided by modulation parameters, and a bidirectional modality guidance mechanism is used for deep feature interaction and fusion. End-to-end optimization is performed by combining cross-entropy loss, triplet loss, and cosine loss to achieve deep complementarity and interaction between modalities.
It improves the stability and discriminativeness of pedestrian re-identification under different lighting conditions, reduces the amount of computation, enhances the consistency and discriminative power of features, and improves all-weather monitoring capabilities.
Smart Images
Figure CN121544995A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of computer vision, and particularly relates to a modal guided fusion method based on visible light and infrared person re-identification. BACKGROUND
[0002] Person Re-Identification (ReID) is a key technology in intelligent video surveillance systems, aiming to retrieve specific pedestrian targets across different non-overlapping camera views. In recent years, visible light single-modal based person re-identification has made significant progress driven by deep learning. However, this method relies heavily on lighting conditions and its performance drops sharply in night or low-light environments, limiting its all-weather application capability. To break this limitation, visible light-infrared cross-modal person re-identification (VI-ReID) emerged. This technology fuses the rich color and texture information of visible light cameras with the characteristics of infrared cameras that are not affected by light, making it possible to achieve all-weather monitoring. Compared with static images, video sequences can provide more rich spatio-temporal context information and dynamic motion features, so the VVI-ReID task based on video has received widespread attention in recent years.
[0003] However, in the video person re-identification of visible light and infrared modalities, the discriminative information contained in different modalities is complementary. The visible light modality is more likely to capture clothing color and texture information, while the infrared modality can better preserve contour and shape features in low-light or night scenes. Existing technologies introduce carved data and skeleton data to narrow the modal difference, greatly increasing the computational load. In addition, some related researches only focus on the timing features of the two modalities, without considering the differences between modalities. SUMMARY
[0004] The application provides a modal guided fusion method based on visible light and infrared person re-identification to solve the above technical problems.
[0005] To solve the above technical problems, one technical solution adopted by the application is: a modal guided fusion method based on visible light and infrared person re-identification, comprising:
[0006] S1. Based on the ResNet50 model, frame-level spatial features of visible light video sequences and infrared video sequences are extracted respectively to obtain visible light modality features and infrared light modality features .
[0007] S2. Color-related information is extracted from the visible light modality to generate channel-level modulation parameters [γ,β], thereby guiding the adaptive enhancement of infrared features to obtain modulated feature vectors Spatial features are fed into a four-layer transformer network to obtain the first visible light temporal features. and first infrared timing features ;
[0008] S3. Based on the first visible light temporal features and the first infrared temporal features, deep feature interaction and fusion are performed through a bidirectional modal guidance mechanism to obtain the enhanced second visible light sequence features. Second infrared sequence features Thus, the cosine loss is obtained. S4. Based on the features of the second visible light sequence and the second infrared sequence, ResNet50 is optimized end-to-end using a joint loss function that combines cross-entropy loss, triplet loss and cosine loss to obtain a modality fusion model.
[0009] Furthermore, the method in step S1 includes:
[0010] S11. Obtain the initial vector of visible light based on video data. and the initial vector of infrared light ;
[0011] S12. Input the visible light initial vector and the infrared light initial vector into two ResNet-50 networks that do not share weights, and then pass them through AP (average pooling operation) to obtain the infrared light... With visible light mode eigenvectors
[0012] Furthermore, the method in step S2 includes:
[0013] S21. Extract color-related information from the visible light modes to generate channel-level modulation parameters [γ,β], thereby guiding the adaptive enhancement of infrared features and obtaining the modulated feature vector.
[0014] S22 feeds the spatial features of infrared and visible light modes into a four-layer transformer network to obtain the first visible light temporal feature. With the first infrared timing features .
[0015] Furthermore, the method in step S3 includes:
[0016] S31. Based on the feature vector as coefficient to guide the fusion, the contribution is dynamically adjusted between different modes to obtain a more balanced and discriminative representation, and to obtain the second visible light sequence feature and the second infrared sequence feature;
[0017] S32. Based on the alignment loss function and fusion vector of cosine similarity, the dominant effect of a single modality is suppressed, and the cosine loss is obtained.
[0018] Furthermore, the method in step S21 includes:
[0019] Based on formulas (1)-(2), the channel coefficients are obtained; where formulas (1)-(2) are:
[0020] (1);
[0021] (2);
[0022] in, For channel coefficients, This is the scaling factor. For bias terms, For characteristic linear modulation; First visible light temporal characteristics; This is the first infrared timing feature; This refers to the infrared features obtained after fusing visible light color features; mean is the average value function, which calculates the mean across the feature dimensions to find the center value of the feature.
[0023] Furthermore, the method in step S22 includes:
[0024] Based on formulas (3)-(4), the second visible light temporal features are obtained. Second infrared timing features Among them, formulas (3)-(4) are:
[0025] (3);
[0026] (4);
[0027] Where LN() represents layer normalization, softmax is the activation function, and... It is a scaling factor used to stabilize softmax calculations. For a Transformer with a depth of 4 layers, LN layers refer to layer normalization.
[0028] Furthermore, the method in step S31 includes:
[0029] Based on formulas (5)-(8), the second visible light sequence features and the second infrared sequence features , Among them, formulas (5)-(8) are:
[0030] The formula is as follows:
[0031] (5);
[0032] (6);
[0033] (7);
[0034] (8);
[0035] Here, sigmoid is the sigmoid activation function, used to improve the model's nonlinearity; liner is a fully connected layer; and cat performs concatenation operations in the batch dimension.
[0036] Furthermore, the method in step S32 includes:
[0037] (9);
[0038] Where • represents the product operation.
[0039] Furthermore, the method in step S3 includes:
[0040] S31. Based on cross-entropy loss and triplet loss, constrain the features of the second visible light sequence and the second infrared sequence to obtain the first function loss and the second function loss;
[0041] S32. Based on the first loss function, the second loss function, and the cosine loss function, obtain the total loss function.
[0042] The beneficial effects of this application are as follows: This application utilizes a bidirectional modality guidance mechanism to enhance infrared feature representation using the temporal and color features of the visible light modality, and optimizes the visible light feature representation using the temporal features of the infrared modality, thereby achieving deep complementarity and interaction between the two modalities. To address the issue of inconsistent feature dimensions, a gating mechanism is introduced to adaptively control the fusion process, ensuring the consistency and discriminative power of the fused features. Furthermore, a cosine loss function is innovatively used to measure and reduce modality differences, and cross-entropy loss and triplet loss are combined to jointly optimize network training. Attached Figure Description
[0043] Figure 1 This is a flowchart illustrating an embodiment of the modal guidance fusion method based on visible light and infrared pedestrian re-identification in this application;
[0044] Figure 2 yes Figure 1 A flowchart illustrating step S1 of an embodiment;
[0045] Figure 3 yes Figure 1A flowchart illustrating step S2 of an embodiment;
[0046] Figure 4 yes Figure 1 A flowchart of an embodiment of step S3.
[0047] Figure 5 yes Figure 1 A flowchart of an embodiment of step S3;
[0048] Figure 6 This is a fusion effect diagram of color information from a visible light image being incorporated into an infrared image in one embodiment of the modality-guided fusion method for pedestrian re-identification based on visible light and infrared in this application.
[0049] Figure 7 This is a kernel density estimation of the positive / negative sample distance distribution in one embodiment of the modality-guided fusion method for pedestrian re-identification based on visible light and infrared light in this application;
[0050] Figure 8 This is a visualization of the cross-modal pedestrian retrieval results in one embodiment of the modal-guided fusion method for pedestrian re-identification based on visible light and infrared light in this application. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments.
[0052] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.
[0053] See Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the modal guidance fusion method based on visible light and infrared pedestrian re-identification according to this application. The method includes:
[0054] S1. Based on the ResNet50 model, frame-level spatial features are extracted from visible light video sequences and infrared video sequences respectively to obtain visible light modal features. and infrared light mode characteristics .
[0055] Among them, see Figure 2 The method of step S1 includes:
[0056] S11. Based on video data, obtain the initial vector of visible light and the initial vector of infrared light.
[0057] Specifically, firstly, the visible light initial vector... The initial vector of infrared light is represented as To align with the RGB input, the infrared frame is copied into three channels in the channel dimension: Where H represents the height of the image, W represents the width of the image, and t represents the image at time t.
[0058] S12. Input the visible light initial vector and the infrared light initial vector into two ResNet-50 networks that do not share weights, and obtain two output feature vectors through AP (average pooling operation). .
[0059] Specifically, the initial vector of visible light and infrared light initial vector The data is fed into the backbones of two ResNet-50 networks that do not share weights. and Obtain the output features. , , is represented as:
[0060] , (5); S2. Extract color-related information from the visible light mode to generate channel-level modulation parameters [γ,β], thereby guiding the adaptive enhancement of infrared features and obtaining the modulated feature vector. Spatial features are fed into a four-layer transformer network to obtain the first visible light temporal features. and first infrared timing features .
[0061] For details, please refer to Figure 3 The method of step S2 includes:
[0062] S21. Extract color-related information from the visible light modes to generate channel-level modulation parameters [γ,β], thereby guiding the adaptive enhancement of infrared features and obtaining the modulated feature vector.
[0063] This mechanism is used to extract color-related information from the visible light modes to generate channel coefficients [γ,β], thereby guiding the adaptive enhancement of infrared light mode features, as expressed below:
[0064] (1);
[0065] (2);
[0066] in, For channel coefficients, This is the scaling factor. For bias terms, For characteristic linear modulation; First visible light temporal characteristics; This is the first infrared timing feature; This refers to the infrared features obtained after fusing visible light color features; mean is the average value function, which calculates the mean across the feature dimensions to find the center value of the feature.
[0067] S22. Subsequently, the temporal features of infrared and visible light are extracted respectively to obtain the first visible light sequence feature f2 and the first infrared sequence feature f3, as shown in the following formula:
[0068] (3);
[0069] (4);
[0070] Here, LN() represents layer normalization, and alpha is a scaling factor used to stabilize softmax computation.
[0071] S3. Based on the first visible light temporal features and the first infrared temporal features, deep feature interaction and fusion are performed through a bidirectional modal guidance mechanism to obtain the enhanced second visible light sequence features. Second infrared sequence features Finally by , Obtained from cosine loss ;
[0072] For details, please refer to Figure 4 The method of step S3 includes:
[0073] S31. Based on the first visible light temporal features and the first infrared temporal features as guiding coefficients, the contribution is dynamically adjusted among different modes. The network mainly extracts the temporal features of the infrared and visible light modes to construct their respective representational bases, and then fuses the unique information of the other mode. The role of this adjustment mechanism is to prevent the model from developing mode bias in the early stage of training, thereby improving the fusion effect and generalization ability, and obtaining the fusion vector.
[0074] Specifically, typical cross-modal fusion strategies often rely on simple element-wise linear superposition, but this strategy lacks modeling of modal differences, resulting in statistical misalignment in the feature space after fusion.
[0075] Specifically, the differences in appearance attributes and temporal dynamics between infrared and visible light can cause one modality to dominate the fused representation, thereby weakening the contribution of the other modality. To overcome this problem, this application does not use linear superposition, but instead uses the previously learned feature vectors as coefficients to guide the fusion, dynamically adjusting the contribution between different modalities to obtain a more balanced and discriminative representation.
[0076] Since directly adding modal features for feature fusion can lead to inconsistencies between the fused infrared and visible light features, this application employs gate control for feature fusion, as shown in the following formula:
[0077] (5);
[0078] (6);
[0079] (7);
[0080] (8);
[0081] Here, sigmoid is the sigmoid activation function, used to improve the model's nonlinearity; liner is a fully connected layer; and cat performs concatenation operations in the batch dimension.
[0082] S32. Alignment loss function based on cosine similarity and fusion vector to suppress the dominant effect of single modality.
[0083] Specifically, traditional methods often lack explicit constraints on intermodal consistency. To reduce the modal gap, an alignment loss based on cosine similarity is further introduced. By forcing consistency between infrared and visible light features in direction, the dominance effect of a single modality is suppressed, and the stability and generalization ability of cross-modal retrieval are improved.
[0084] The cosine loss function is used to measure the difference between infrared and visible light modes, and its formula is defined as follows:
[0085] (9);
[0086] Where • represents the product operation. S3. Based on the second visible light sequence features and the second infrared sequence features, the ResNet50 is optimized end-to-end using a joint loss function that combines cross-entropy loss, triplet loss, and cosine loss to obtain a modality fusion model;
[0087] Specifically, the method in step S3 includes:
[0088] S31. Based on cross-entropy loss and triplet loss, constraints are applied to the second visible light sequence features and the second infrared sequence features to obtain the first loss function and the second loss function, where the specific formulas are as follows:
[0089] Finally, the first loss function of the fusion module is designed by combining cross-entropy and triples. The formula is as follows:
[0090] (10);
[0091] (11);
[0092] (12);
[0093] in, For L2norm, It is a fully connected layer. Here, is the label of the i-th person, and b is the number of batches sent. Represents positive samples. This represents a negative sample. Finally, the second loss function of the baseline. It is expressed as follows:
[0094] (13);
[0095] S32. Based on the first loss function and the second loss function, obtain the total loss function; the specific formula is as follows:
[0096] (14).
[0097] like Figure 6 The diagram illustrates the fusion of infrared and visible light color features. The first row shows the infrared image, the second row shows the visible light image, and the third row shows the result of incorporating color features into the infrared image. Visible light color enhancement makes the infrared image clearer in detail while preserving its structure.
[0098] In cross-modal pedestrian retrieval tasks, given a visible light query image, the model returns a ranked list from high to low similarity based on feature similarity in an infrared image gallery. As shown in the upper part of Figure 8, using a visible light image as the query sample, the model retrieves the most similar images from multiple infrared images and sorts them from left to right. Rank-1 accuracy refers to the percentage of correct retrievals if the first image in the ranked list (the leftmost image) matches the query sample. Rank-1 represents the proportion of correct retrievals.
[0099] Furthermore, to verify the necessity of scale alignment (Equation 7-10) during the fusion of infrared and visible light features, this method was compared on HITSZ-VCM, and the results are shown in Table 1. Here, "baseline" represents the ResNet50 network, and "temporal features" represents the temporal features utilizing only the infrared and visible light modes. The color features indicate that the infrared mode incorporates the color features of visible light. Alignment represents the introduction of gate control (IR_gate and VIS_gate), and alignment loss represents the cosine loss function. In our method, infrared features are fused with color and temporal information from visible light, while visible light features are fused only with the temporal information of infrared. This asymmetric fusion method can easily lead to differences in numerical distribution and scale between the two modes, causing one mode to dominate the fused representation. Experimental results show that introducing scale alignment can effectively balance the feature contributions of the two modes, making the fused representation more stable and more discriminative. Furthermore, comparisons with other state-of-the-art methods in Table 2 demonstrate the state-of-the-art nature of our model.
[0100] Table 1 shows the ablation experimental diagrams of this fusion method.
[0101]
[0102] Table 2 Comparison of different methods
[0103]
[0104] To verify the effectiveness of the method from a statistical distribution perspective, a histogram of feature distance distribution was plotted, as shown in Figure 7. The horizontal axis represents the Euclidean distance between sample pairs, and the vertical axis represents the corresponding probability density. The blue curve represents sample pairs with the same identity (within class), and the orange curve represents sample pairs with different identities (between class). It can be observed that under the baseline model (ResNet50), the interval between the within-class and between-class distributions is relatively small (…). =0.271), indicating significant overlap; however, after introducing this method, the intra-class distribution becomes more compact, the inter-class distribution becomes more dispersed, and the interval increases significantly ( =0.361). This indicates that the proposed method can effectively reduce the feature distance of the same identity while expanding the feature interval between different identities, thereby improving cross-modal discriminability.
[0105] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A modal guidance fusion method based on visible light and infrared pedestrian re-identification, characterized in that, include: S1. Based on the ResNet50 model, frame-level spatial features of visible light video sequences and infrared video sequences are extracted respectively to obtain visible light modal features and infrared light modal features; S2. Extract color-related information from the visible light mode to generate channel-level modulation parameters and obtain the modulated feature vector; feed the spatial features into a four-layer transformer network to obtain the first visible light temporal features and the first infrared temporal features; S3. Based on the first visible light temporal features and the first infrared temporal features, deep feature interaction and fusion are performed through a bidirectional modal guidance mechanism to obtain enhanced second visible light sequence features and second infrared sequence features, thereby obtaining cosine loss; S4. Based on the second visible light sequence features and the second infrared sequence features, the ResNet50 is optimized end-to-end using a joint loss function that combines cross-entropy loss, triplet loss and cosine loss to obtain a modality fusion model.
2. The method according to claim 1, characterized in that, The method of step S1 includes: S11. Based on video data, obtain the initial vectors for visible light and infrared light; S12. Input the visible light initial vector and the infrared light initial vector into two ResNet-50 networks that do not share weights and perform average pooling to obtain the infrared light and visible light modal feature vectors.
3. The method according to claim 1, characterized in that, The method of step S2 includes: S21. Extract color-related information from the visible light modes to generate channel-level modulation parameters, thereby guiding the adaptive enhancement of infrared features and obtaining the modulated feature vector; S22 feeds the vector space features of infrared light mode features and visible light mode features into a four-layer transformer network to obtain the first visible light temporal features and the first infrared temporal features.
4. The method according to claim 3, characterized in that, The method of step S3, S31. Based on the feature vector as coefficients to guide fusion, dynamically adjust the contribution between different modes to obtain a more balanced and discriminative representation, and obtain the second visible light sequence feature and the second infrared sequence feature; S32. Based on the alignment loss function and fusion vector of cosine similarity, the dominant effect of a single modality is suppressed, and the cosine loss is obtained.
5. The method according to claim 3, characterized in that, The method of step S21 includes: Based on formulas (1)-(2), the channel coefficients are obtained; wherein, formulas (1)-(2) are: （1）； （2）； in, For channel coefficients, This is the scaling factor. For bias terms, For characteristic linear modulation; First visible light temporal characteristics; This is the first infrared timing feature; This refers to the infrared features obtained after fusing visible light color features; mean is the average value function, which calculates the mean across the feature dimensions to find the center value of the feature.
6. The method according to claim 5, characterized in that, The method of step S22 includes: Based on formulas (3)-(4), the second visible light temporal features are obtained. and the second infrared timing feature ; where formulas (3)-(4) are: （3）； （4）； Where LN() represents layer normalization, softmax is the activation function, and... It is a scaling factor used to stabilize softmax calculations. For a Transformer with a depth of 4 layers, LN layers refer to layer normalization.
7. The method according to claim 4, characterized in that, The method of step S31 includes: Based on formulas (5)-(8), the second visible light sequence feature and the second infrared sequence feature , ; where formulas (5)-(8) are: The formula is as follows: （5）； （6）； （7）； （8）； Here, sigmoid is the sigmoid activation function, used to improve the model's nonlinearity; liner is a fully connected layer; and cat performs concatenation operations in the batch dimension.
8. The method according to claim 4, characterized in that, The method of step S32 includes: Based on formula (9), the cosine loss is obtained; wherein, the formula （9）； Where • represents the product operation.
9. The method according to claim 1, characterized in that, The method of step S3 includes: S31. Based on cross-entropy loss and triplet loss, constrain the second visible light sequence features and the second infrared sequence features to obtain the first function loss and the second function loss; S32. Based on the first loss function, the second loss function, and the cosine loss function, obtain the total loss function.