A cross-modal pedestrian re-identification method based on intermediate feature guidance

Abstract

The present application belongs to the field of computer vision, and particularly relates to a cross-modal pedestrian re-identification method based on intermediate feature guidance. The present application method takes ResNet50 as a backbone network, proposes a novel feature mining and fusion module, and according to the phenomenon that the multi-modal data of the cross-modal pedestrian re-identification has a large bias, proposes to use the intermediate modal to reduce the modal gap, and proposes a three-modal three-tuple loss function and a center bias loss function, which jointly train the network with the common id loss function, effectively improving the recognition accuracy of the model. The present application proposes to use the intermediate feature containing two original modalities to alleviate the modal difference, effectively improving the recognition accuracy of the model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a cross-modal person re-identification method based on intermediate features. Background Technology

[0002] With the rapid development of society and the economy, public safety has become increasingly important. Today, surveillance equipment is ubiquitous in daily life; however, the application of surveillance information remains limited to a relatively basic level. In emergency situations, manually reviewing surveillance footage to identify and retrieve suspects is a time-consuming and labor-intensive task, especially in complex shooting environments where the accuracy of human visual perception is relatively low.

[0003] Therefore, the industry has begun to actively explore Person Re-identification (Re-ID) technology, which has become an important part of the intelligent security field. Person re-identification is a technology designed to identify specific pedestrians from a large number of pedestrian images captured by different cameras to meet various missing person needs. The task involves matching images or videos of related individuals captured by multiple non-overlapping cameras. Due to its wide range of practical applications, such as criminal investigation, danger warning, unmanned supermarket management, and missing person rescue, this task has become crucial. Person re-identification has become a hot topic in scientific research and has received widespread attention.

[0004] Early pedestrian re-identification technologies were limited to processing visible light pedestrian images. However, in practical applications, relying solely on a single modality cannot meet the diverse needs of various scenarios. For example, some criminals often commit crimes at night or choose locations with poor lighting conditions to hide or flee. In poor lighting conditions, visible light cameras struggle to capture clear pedestrian images, thus significantly reducing the accuracy of single-modality pedestrian re-identification technology. To address this issue, surveillance equipment has been upgraded. The new generation of surveillance equipment incorporates both visible light and infrared modal cameras, automatically switching shooting modes based on lighting conditions, thereby overcoming the limitations imposed by lighting conditions on identification. This technological upgrade makes surveillance systems more adaptable to different environments and situations, significantly improving the reliability of identification.

[0005] However, this upgrade also brings new challenges, namely the need to process images of two different modalities simultaneously, which significantly increases the complexity of the original single-modal person re-identification task. Therefore, in order to ensure that person re-identification technology can function in various scenarios, the academic community has begun to study the "visible-infrared" cross-modal person re-identification task (VI-ReID).

[0006] Visible-infrared (VI) person re-identification is a more challenging variant of the person re-identification (ReID) task, involving image matching across visible light (RGB) and infrared (IR) cameras. The challenge of VI ReID lies in matching individuals with significant appearance differences between the two different modalities. For example, visible light images capture color information, while infrared images provide information about body temperature and are less affected by variations in lighting conditions. State-of-the-art VI ReID systems typically leverage backbone models such as convolutional neural networks (CNNs) or visual transformers (ViTs) to extract features from both visible light and infrared images, then match these features before cross-modal feature fusion.

[0007] The SYSU-MM01 dataset was created in the 2017 paper "RGB-infrared cross-modality person re-identification". SYSU-MM01 is the first dataset specifically designed for cross-modal person re-identification tasks. This dataset includes pedestrian images captured by indoor and outdoor visible light cameras, as well as images captured by infrared cameras. The training set contains 395 different pedestrian identities, comprising 22,258 visible light images and 11,909 infrared images. The test set consists of images of 96 different pedestrians and allows for two different evaluation modes: All-Search and Indoor-Search.

[0008] In recent years, many cross-modal pedestrian re-identification methods have been proposed.

[0009] For example, the paper "Visible-infraredperson re-identification using privileged intermediate information" noted the excessive modality gap and therefore addressed it by aligning features to reduce it. This approach utilizes multiple modules to extract fine-grained features, enhancing cross-modal discriminative ability, and then aligns features across different modalities. A modality elimination module removes modality-specific information, allowing the model to focus more on shared identity features across modalities. A centroid clustering loss optimizes identity learning and detailed feature learning, improving feature discriminative power. Figure 9 As shown.

[0010] In contrast, while this paper acknowledges the modal differences between different modalities, it only uses the original two modalities for cross-modal information alignment. This approach is difficult to implement and hinders the mining of shared pedestrian information across different modalities, significantly impacting its accuracy. Existing pedestrian re-identification methods ignore the modal differences between different modalities, which is detrimental to mining shared pedestrian information across modalities and will significantly affect the accuracy of cross-modal pedestrian re-identification. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the prior art and propose a cross-modal pedestrian re-identification method based on intermediate features, thereby improving the accuracy of model recognition.

[0012] The technical solution adopted in this invention is as follows: A cross-modal person re-identification method based on intermediate features, comprising the following steps:

[0013] Step 1: Perform data preprocessing on P*K visible light and infrared images, then proceed to Step 2;

[0014] Step 2: Input the P*K preprocessed visible light images and P*K infrared images obtained in Step 1 into the feature extraction network, and proceed to Step 3;

[0015] Step 3: Using the ResNet50 network and feature mining module, generate information-rich image features based on the image input in Step 2, and proceed to Step 4;

[0016] Step 4: Perform dot product fusion on the image features of the two branches to form an intermediate feature. Input the intermediate feature together with the original features of the two branches into the feature fusion module to obtain a new intermediate feature, and proceed to step 5.

[0017] Step 5: Input the image features and intermediate features of the two branches obtained in Step 3 and Step 4 into the loss function, including ID loss, trimodal triplet loss function and center bias loss function, so as to train a network with good performance and proceed to Step 6;

[0018] Step 6: If the specified number of training rounds is reached, proceed to Step 7; otherwise, continue training and return to Step 1.

[0019] Step 7: End.

[0020] Furthermore, the steps for extracting preliminary image features in step 2 are as follows:

[0021] Step 2-1: Input the visible light image into the feature extraction network to extract its preliminary features;

[0022] Step 2-2: Input the infrared image into the feature extraction network to extract its preliminary features;

[0023] Steps 2-3: Output features.

[0024] Furthermore, in step 2, the first layer of the residual network is used for feature extraction to fully perceive the shallow information of the image, and the feature extraction network parameters of the visible light image and the infrared image are not shared.

[0025] Furthermore, the steps for extracting image features in step 3 are as follows:

[0026] Step 3-1: Input the features extracted in Step 2 into a network consisting of the last four convolutional layers of ResNet50 and attention blocks of the feature mining module to obtain information-rich image features;

[0027] Step 3-2: Perform MAX-pooling on the features from Step 3-1, then proceed to Step 3-3;

[0028] Step 3-3: Output the pooling features of the image.

[0029] Furthermore, in step 3, the feature mining module introduces a spatial attention module and a dimensional attention module. The spatial attention module focuses on the dependency between local features and spatial location to enhance the perception of the target region; while the dimensional attention module focuses on channel information, adjusting the importance of different feature channels so that the network can dynamically allocate weights and improve the expressive power of key features.

[0030] The input feature f0 is input into two different attention modules respectively. Spatial attention and dimensional attention are used to obtain attention features that emphasize different information. Then, a dot product operation is performed with the input feature, and activation and batch normalization are performed. The features of the two branches are then multiplied to obtain the feature f1 that integrates spatial and dimensional attention.

[0031] For the obtained feature f1, global max pooling and global average pooling are performed respectively, and the feature is concatenated to obtain a feature vector with a dimension of 2. In order to further enhance the discriminative ability of the model, matrix multiplication is performed with the feature that integrates spatial and dimensional attention, so that the output feature can fully integrate information from different attention mechanisms, and finally obtains a representation feature f2 with the same dimension as the original input feature but richer.

[0032] Finally, during the training of deep networks, the vanishing and exploding gradient problems can lead to unstable model convergence. To avoid gradient descent, a residual connection mechanism is introduced in the final feature generation process, which involves adding f2 to the original input features element by element.

[0033] Furthermore, in step 4, the intermediate features, along with the features from the original two branches, are input into the feature fusion module to obtain new intermediate features, achieving the goal of deep fusion of the two modal features. The specific steps are as follows:

[0034] Step 4-1: Extract the visible light image features F from Step 3. vi Intermediate feature F after transpose mid Perform matrix multiplication, then pass through ReLU activation, and finally combine with the visible light image features F. vi Perform matrix multiplication, then combine with the intermediate feature F mid Perform bitwise addition to obtain F fused1 ;

[0035] Step 4-2: Extract the infrared image features F from Step 3. in The input is fed into the SelfAttention network to obtain feature F. in1 ; feature F in1 With the transposed feature F fused1 Perform matrix multiplication, then pass through ReLU activation, and then combine with feature F again. in1 Perform matrix multiplication, then combine with the intermediate feature F fused1 Perform bitwise addition to obtain the characteristic F. fused2 ;

[0036] Step 4-3: Transfer feature F fused2 The input is fed into the SelfAttention network to obtain the output feature F. fused ;

[0037] Step 4-4: Output feature F fused Perform BN normalization.

[0038] Furthermore, in step 5, the id loss function is used to jointly constrain the model. The specific steps are as follows:

[0039] Step 5-1: Pool the features extracted in Step 3;

[0040] Step 5-2: Perform Batch Normalization on the features extracted in Step 5-1;

[0041] Step 5-3: Pass the normalized features from Step 5-2 through a fully connected layer to reduce the dimensionality of the information while obtaining the dimensional information corresponding to the category, and proceed to Step 5-4;

[0042] Step 5-4: Then input the features into the id loss function;

[0043] Step 5-5: Using the losses from 4-4 and 5-4, calculate the gradient, perform backpropagation, and update the model parameters;

[0044] During training, the ID loss is calculated by selecting P identities from the training set and randomly selecting K pedestrian images from each identity. Each batch contains 2*P*K pedestrian images. The identity loss for the pedestrian re-identification task is shown in the following formula:

[0045]

[0046] Among them, y i,k p represents whether the identity of the i-th image is k, N represents the total number of pedestrian categories in the dataset, and p i,k Let represent the probability that the identity of the i-th image is k.

[0047] Furthermore, in step 5, the trimodal triplet loss function uses the image features of the intermediate modality as anchor points to bring the farthest images of the same identity closer together and push away the closest images of different identities for the two original modal images respectively.

[0048] For the extracted trimodal data, it is necessary to make full use of the generated intermediate modality features to achieve more efficient pedestrian identification. To this end, a trimodal triplet loss is specifically proposed. The common hard triplet loss is a widely used loss function in pedestrian re-identification tasks. It brings the furthest positive pairs closer together and pushes the nearest negative pairs further apart. Simply put, it brings the feature distances of pedestrian images of the same type of person taken from different cameras closer together, and pushes the feature distances of different pedestrian images further apart.

[0049] The existing difficult cross-modal triplet loss, i.e., the difficult two-modal triplet loss;

[0050] The same shape represents images of the same category, i.e., different modalities of the same pedestrian's identity; different shapes represent images of different categories, i.e., different pedestrians' identities. For each image in each batch, the distance to it is calculated, and the positive sample farthest from the image and the negative sample closest to the image are selected to calculate the loss, as shown in Equation (2):

[0051]

[0052] In the formula, A is the anchor sample, (A,p) is the sample with the same identity category as the anchor sample, and (A,n) is the sample with a different identity category than the anchor sample. For the threshold parameter, use Calculate the Euclidean distance, representing the characteristic f. i with f i The Euclidean distance is calculated using the formula shown in equation (3):

[0053]

[0054] The trimodal triplet loss function uses the image features of the intermediate modality as anchor points, and pulls the farthest image of the same identity closer and pushes the nearest image of different identities further away for the two original modal images respectively.

[0055] For each intermediate feature sample in each batch, it is used as an anchor sample, and the loss is calculated by comparing it with the positive sample farthest from the original modality image and the negative sample closest to the original modality image, as shown in Equation (4):

[0056]

[0057] In the formula, A is the anchor sample, (A,p) is the positive sample, (A,n) is the negative sample, and ε is the threshold parameter.

[0058] Furthermore, in step 5, the center deviation loss function takes into account the strong expressive power of visible light images and the fact that the ResNet50 network parameters are pre-trained with visible light images, which may lead to a strong visible light image bias in the model. In order to avoid the intermediate mode being affected by this factor, the center deviation loss function will be constrained for the feature center design of the three modes.

[0059] First, for each type of pedestrian in the three modalities, the feature center is calculated, and the intermediate feature is constrained to be at the center of the two original features. The intermediate feature is also constrained to be equidistant from the two original features.

[0060] For each unique ID image in each batch, the feature center for that category is calculated separately, and then the visible light feature F is obtained. vi With infrared feature F i The feature center is determined, and the distance between it and the intermediate feature center is calculated using the formulas shown in equations (5) and (6):

[0061]

[0062]

[0063] In the formula, Batch i With Batch vi These represent the visible light and infrared features in the batch, respectively; K represents the number of images for each modality of each sample in the batch; P represents the number of sample categories in the batch; and c... final_i c represents the center of the i-th intermediate feature. mid_i It is the center representing the visible light feature center and the infrared feature center of the i-th category;

[0064] The total network loss is then expressed as shown in formula (7):

[0065] L all =Lid +λ1L Tri_Tri_Loss +λ2L Center_Bias_Loss (7)

[0066] λ1 and λ2 are hyperparameters that are set separately in the experiment.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] (1) This invention uses ResNet50 as the backbone network, proposes a novel feature mining and fusion module, and proposes to use intermediate modes to reduce the modality gap based on the phenomenon of large deviation of multimodal data in cross-modal pedestrian re-identification. It also proposes a trimodal triplet loss function and a center deviation loss function, and trains the network together with the common id loss function, which effectively improves the recognition accuracy of the model.

[0069] (2) This invention proposes to use intermediate features that simultaneously contain two original modalities to alleviate modal differences and effectively improve the recognition accuracy of the model. Attached Figure Description

[0070] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0071] Figure 1 These are pedestrian images in color and infrared modes from the SYSU-MM01 dataset used in this invention.

[0072] Figure 2 This is a schematic diagram of the pedestrian re-identification network framework of the present invention;

[0073] Figure 3 This is a flowchart of the training phase of the present invention;

[0074] Figure 4 This is a structural diagram of the feature fusion module of the present invention;

[0075] Figure 5 This is a schematic diagram of the cross-modal triplet of the present invention;

[0076] Figure 6 This is a schematic diagram of the three-modal triplet loss function of the present invention;

[0077] Figure 7 This is a schematic diagram of the modal deviation of the present invention;

[0078] Figure 8 This is a structural diagram of the feature mining module of the present invention;

[0079] Figure 9 This is a schematic diagram of the comparative method of the present invention. Detailed Implementation

[0080] The present invention will be further explained in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand and implement the present invention. However, the following examples are only used to explain the present invention and are not intended to limit the present invention.

[0081] This invention proposes a cross-modal person re-identification method based on intermediate features. The framework diagram of the network is as follows: Figure 2 As shown.

[0082] The training flowchart of the network proposed in this invention is as follows: Figure 3 As shown. This training process uses a mini-batch training method. In each batch, P pedestrians are randomly selected, and K visible light and infrared images are randomly selected for each of these pedestrians, resulting in a total of 2P*K pedestrian images. The training process will now be described using a single input image as an example, as follows:

[0083] Step 1: Perform data preprocessing on P*K visible light and infrared images, then proceed to Step 2.

[0084] Step 2: Input the P*K preprocessed visible light images and P*K infrared images obtained in Step 1 into the feature extraction network, and proceed to Step 3;

[0085] Step 3: Using the ResNet50 network and the feature mining modules with the third and fourth layers of the ResNet50 network, generate information-rich image features based on the image input in Step 2, and proceed to Step 4;

[0086] Step 4: Perform dot product fusion on the image features of the two branches to form an intermediate feature. Input the intermediate feature together with the original image features of the two branches into the feature fusion module to obtain a new intermediate feature, and proceed to Step 5.

[0087] Step 5: Input the image features and intermediate features from the two branches obtained in Steps 3 and 4 into the loss functions, including ID loss, trimodal triplet loss function, and center bias loss function. This trains a high-performance network, leading to Step 6.

[0088] Step 6: If the specified number of training rounds is reached, proceed to Step 7; otherwise, continue training and return to Step 1.

[0089] Step 7: End.

[0090] The images input in step 1 all considered three data preprocessing methods: random cropping, horizontal flipping, and erasing.

[0091] The steps for extracting preliminary image features in step 2 of the training process in this invention are as follows:

[0092] Step 2-1: Input the visible light image into the feature extraction network to extract its preliminary features;

[0093] Step 2-2: Input the infrared image into the feature extraction network to extract its preliminary features;

[0094] Steps 2-3: Output features.

[0095] In step 2, this invention utilizes the first layer of a residual network for feature extraction, fully perceiving the shallow information of the image, and the feature extraction network parameters for visible light images and infrared images are not shared.

[0096] The steps for extracting image features in step 3 of the training process in this invention are as follows:

[0097] Step 3-1: Input the features extracted in Step 2 into a network consisting of the last four convolutional layers of ResNet50 and attention blocks of the feature mining module to obtain information-rich image features;

[0098] Step 3-2: Perform MAX-pooling on the features from Step 3-1, then proceed to Step 3-3;

[0099] Step 3-3: Output the pooling features of the image.

[0100] In step 3, this invention utilizes a residual network and a feature mining module to extract features. Unlike the feature extraction network in step 2, parameters are shared in this stage. In the feature mining module proposed in this invention, spatial attention and channel attention are introduced to fully utilize feature information and improve the model's discriminative ability and generalization performance. Its structure is as follows: Figure 8 As shown.

[0101] The Spatial Attention Module (SAM) focuses on the dependencies between local features and spatial locations to enhance the perception of target regions. Meanwhile, the Dimensional Attention Module (DAM) focuses on channel information, adjusting the importance of different feature channels, enabling the network to dynamically allocate weights and improve the expressive power of key features.

[0102] The input feature f0 is fed into two different attention modules, utilizing spatial attention and dimensional attention to obtain attention features that emphasize different information. These features are then multiplied with the input feature, followed by activation and batch normalization. Finally, the features from both branches are multiplied again to obtain the feature f1, which combines spatial and dimensional attention.

[0103] For the obtained feature f1, global max pooling and global average pooling are performed respectively, and the feature is concatenated to obtain a feature vector with a dimension of 2. In order to further enhance the discriminative ability of the model, matrix multiplication is performed with the feature that integrates spatial and dimensional attention, so that the output feature can fully integrate information from different attention mechanisms, and finally obtains a representation feature f2 with the same dimension as the original input feature but richer.

[0104] Finally, during deep network training, the vanishing and exploding gradient problems can lead to unstable model convergence. To avoid gradient descent, a residual connection mechanism is introduced in the final feature generation process, which involves adding f2 element-wise with the original input features.

[0105] In step 4, the present invention proposes a feature fusion module, the structure of which is as follows: Figure 4 As shown, the intermediate features, along with the features from the original two branches, are input into the feature fusion module to obtain new intermediate features, achieving the goal of deep fusion of the intermediate features with the two modal features. The specific steps are as follows.

[0106] Step 4-1: Extract the visible light image features F from Step 3. vi Intermediate feature F after transpose mid Perform matrix multiplication, then pass through ReLU activation, and finally combine with the visible light image features F. vi Perform matrix multiplication, then combine with the intermediate feature F mid Perform bitwise addition to obtain F fused1 ;

[0107] Step 4-2: Extract the infrared image features F from Step 3. in The input is fed into a self-attention network to obtain feature F. in1 . The feature F in1 With the transposed feature F fused1 Perform matrix multiplication, then pass through ReLU activation, and then combine with feature F again. in1 Perform matrix multiplication, then combine with the intermediate feature F fused1 Perform bitwise addition to obtain the characteristic F. fused2 ;

[0108] Step 4-3: Transfer feature F fused2 The input is fed into a self-attention network to obtain the output feature F. fused ;

[0109] Step 4-4: Output feature F fused Perform BN normalization.

[0110] In step 5, the present invention uses the id loss function to jointly constrain the model. The specific steps are as follows.

[0111] Step 5-1: Pool the features extracted in Step 3;

[0112] Step 5-2: Perform BN normalization on the features extracted in Step 5-1.

[0113] Step 5-3: Pass the normalized features from Step 5-2 through a fully connected layer to reduce the dimensionality of the information while obtaining the dimensional information corresponding to the category, and proceed to Step 5-4;

[0114] Step 5-4: Then input the features into the id loss function;

[0115] Step 5-5: Calculate the gradient using the losses from 4-4 and 5-4, perform backpropagation, and update the model parameters.

[0116] Specifically, during training, the ID loss involves selecting P identities from the training set, and randomly selecting K pedestrian images from each identity. Each batch contains 2*P*K pedestrian images. The identity loss for the pedestrian re-identification task is shown in the formula below.

[0117]

[0118] Among them, y i,k p represents whether the identity of the i-th image is k, N represents the total number of pedestrian categories in the dataset, and p i,k This represents the probability that the identity of the i-th image is k.

[0119] In step 5, the present invention uses a trimodal triplet loss function and a central deviation loss function to jointly constrain the model.

[0120] For the extracted trimodal data, it is necessary to make full use of the generated intermediate modality features to achieve more efficient pedestrian identification. To this end, a trimodal triplet loss is specifically proposed. The common hard triplet loss is a widely used loss function in pedestrian re-identification tasks; it brings the furthest positive pairs closer together and pushes the nearest negative pairs further apart. Simply put, it brings the feature distances of pedestrian images of the same type of person taken from different cameras closer together, and pushes the feature distances of different pedestrian images further apart.

[0121] Existing difficult cross-modal triplet loss, i.e. difficult two-modal triplet loss, is illustrated as follows: Figure 5 As shown.

[0122] In the example image, green arrows represent features that are brought closer together, increasing their similarity, while red arrows represent features that are pushed away, decreasing their similarity. The same shape represents images of the same category, i.e., different modalities of the same pedestrian, while different shapes represent images of different categories, i.e., different pedestrians. For each image in each batch, the distance to it is calculated, and the positive sample farthest from the image and the negative sample closest to the image are selected to calculate the loss, as shown in Equation (2):

[0123]

[0124] In the formula, A is the anchor sample, (A,p) is the sample with the same identity category as the anchor sample, and (A,n) is the sample with a different identity category than the anchor sample. For the threshold parameter, use Calculate the Euclidean distance, representing the characteristic f. i with f i The Euclidean distance is calculated using the formula shown in equation (3):

[0125]

[0126] Compared to traditional cross-modal triplet loss, which directly merges or de-merges features between two modalities, the significant modal differences can negatively impact performance. The trimodal triplet loss function, however, uses the image features of the intermediate modality as anchors, merging the furthest images of the same identity from both original modalities and de-merging the closest images of different identities. A schematic diagram is shown below. Figure 6 As shown.

[0127] For each intermediate feature sample in each batch, it is used as an anchor sample, and the loss is calculated by comparing it with the positive sample farthest from the original modality image and the negative sample closest to the original modality image, as shown in Equation (4):

[0128]

[0129] In the formula, A is the anchor sample, (A,p) is the positive sample, (A,n) is the negative sample, and ε is the threshold parameter.

[0130] Considering the strong expressive power of visible light images and the fact that the ResNet50 network parameters are pre-trained using visible light images, the model may exhibit a strong bias towards visible light images. To avoid the intermediate modes being affected by this factor, a center deviation loss function will be designed to constrain the feature centers of the three modes. The concept of modal deviation is as follows: Figure 7 As shown.

[0131] In the diagram, each dashed circle represents the same person in the same modality, and the dark pattern represents the feature center of that person in this batch. Therefore, we first find the feature center for each type of pedestrian in the three modalities, and constrain the intermediate feature to be at the center of the two original features. Furthermore, we also constrain the intermediate feature to be equidistant from the two original features.

[0132] For each unique ID image in each batch, the feature center for that category is calculated separately, and then the visible light feature F is obtained. vi With infrared feature F i The feature center is determined, and the distance between it and the intermediate feature center is calculated using the formulas shown in equations (5) and (6):

[0133]

[0134] In the formula, Batch i With Batch vi These represent the visible light and infrared features in the batch, respectively; K represents the number of images for each modality of each sample in the batch; P represents the number of sample categories in the batch; and c... final_i c represents the center of the i-th intermediate feature. mid_i It is the center representing the visible light feature center and the infrared feature center of the i-th category.

[0135] The total network loss can then be expressed as shown in formula (7).

[0136] L all =L id +λ1L Tri_Tri_Loss +λ2L Center_Bias_Loss (7)

[0137] λ1 and λ2 are hyperparameters that are set separately in the experiment.

[0138] The testing procedure for this invention is as follows:

[0139] Step S1: Input the query set and gallery set, then proceed to step S2;

[0140] Step S2: Using the model obtained during training, extract features from all pedestrian images in the query set and gallery set input in Step S1, and proceed to Step S3;

[0141] Step S3: Calculate the similarity between the query set features and the image library features, then proceed to step S4;

[0142] Step S4: Based on the similarity level, obtain the matching result corresponding to each pedestrian image in the query set, and proceed to step S5;

[0143] Step S5: End.

[0144] In step S1 of the test process, the query set represents the set of pedestrian images to be queried, while the image library set represents the set of pedestrian images that match the query set.

[0145] The similarity calculation method in step S3 of the testing process is dot product similarity.

[0146] Example 1:

[0147] In this embodiment, mini-batch gradient descent is used to update the model parameters of the present invention, that is, a small batch of samples is randomly selected for parameter updates during each gradient descent. A ResNet50 network is used as the basic skeleton, where the parameters of ResNet50 are pre-trained on ImageNet. The stride of the last bottleneck block of the teacher network is set to 1. For image preprocessing, three data augmentation methods are considered: random cropping, horizontal flipping, and erasure.

[0148] In the experiment, the image size was uniformly set to a specific dimension. Furthermore, the momentum parameter was set to 0.9, and the initial learning rate was set to 0.01. A warm-up learning rate strategy was used for the first 10 epochs, and then the learning rate was adjusted to 0.01 and 0.001 in the 20th and 50th epochs, respectively, for a total of 120 training epochs. During the testing phase, pedestrian features processed by batch normalization (BN) layers were used for retrieval. This experiment employed the deep learning framework PyTorch 1.8 and used an NVIDIA 3090 graphics card to accelerate the training process. These parameter and setting choices were designed to optimize model performance and training efficiency.

[0149] like Figure 1 As shown in the figure, this embodiment will use the SYSU-MM01 dataset to complete the pedestrian re-identification task and test the performance of the model to evaluate the method in this paper. The comparison results are shown in Table 1.

[0150] Table 1. Performance comparison of the method of this invention with other methods on the SYSU-MM01 dataset.

[0151]

[0152]

[0153] In experiments on the SYSU-MM01 dataset, compared to other methods, the SFANet method only reduces the feature gap in the spectrum, essentially transforming the data into a format with slightly smaller modal differences. Experimental results show that this approach has limited effectiveness. In contrast, the method presented in this chapter achieves higher recognition accuracy. In full search mode, Rank-1 and mAP are improved by 2.99% and 5.29%, respectively; in indoor search mode, Rank-1 and mAP are improved by 4.16% and 0.38%, respectively. This demonstrates that the method in this chapter, employing a three-stream network structure to extract intermediate features, achieves a more significant effect in narrowing modal features, and the optimization of the feature measurement stage significantly improves the overall performance.

[0154] Compared to the PMWGCN method, which uses noise filtering to attempt to filter cross-modal differences as noise, the infrared modality data itself contains less information. The method in this chapter first extracts modality-specific features, preserving as much valuable pedestrian feature information as possible from the visible light modality. Then, it uses a feature extraction network with an attention mechanism to extract more similar modal features, mitigating cross-modal differences. Therefore, its performance is slightly lower than this method. In full search mode, Rank-1 and mAP are improved by 1.91% and 1.24%, respectively; in indoor search mode, Rank-1 and mAP are improved by 3.12% and 5.04%, respectively.

[0155] Example 2:

[0156] In large shopping malls, multiple surveillance cameras are typically deployed to enhance the customer experience, optimize mall management, and ensure public safety. However, the mall environment is characterized by significant variations in lighting conditions, such as strong light at entrances, complex changes in indoor lighting, and low-light areas in some corners. Traditional single visible light cameras are insufficient to meet the needs of 24 / 7 pedestrian tracking. Therefore, combining infrared and visible light cameras and utilizing cross-modal pedestrian re-identification technology can achieve accurate pedestrian matching and behavior analysis.

[0157] First, the system collects video streams from surveillance cameras in different areas of the mall and extracts pedestrian images using a pedestrian detection algorithm. These images are accompanied by camera numbers and timestamps for subsequent analysis.

[0158] These visible light and infrared images are then input into the cross-modal recognition model. The core task of this model is to identify the image of the same customer under different lighting conditions and match their trajectory throughout the entire area. For example, if a customer is captured by a visible light camera at the entrance of a shopping mall and then recorded by an infrared camera in a low-light area or parking lot, the model can accurately match these images, thereby achieving cross-modal re-identification.

[0159] Next, the system analyzes customers' movement trajectories based on the matching results, examining their shopping habits such as frequently visited areas, dwell time, and frequently visited stores. This information can be used to optimize store layout, adjust merchandise placement strategies, and provide personalized recommendations. Furthermore, in terms of abnormal behavior detection, this technology can also be used to identify individuals who linger for extended periods, exhibit unusual activity, or may pose a theft risk, allowing mall management to intervene promptly.

[0160] Ultimately, cross-modal pedestrian re-identification technology not only improves the intelligent management level of shopping malls, but also enhances public safety, ensuring that consumers' shopping experience is more targeted and safer.

[0161] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A cross-modal person re-identification method based on intermediate features, characterized in that, Includes the following steps: Step 1: For After performing data preprocessing on the visible light and infrared images, proceed to step 2; Step 2: Process the preprocessed data obtained in Step 1. Zhang Visible Light Images and The infrared image is input into the feature extraction network, and the process proceeds to step 3; Step 3: Using the ResNet50 network and feature mining module, generate information-rich image features based on the image input in Step 2, and proceed to Step 4; Step 4: Perform dot product fusion on the image features of the two branches to form an intermediate feature. Input the intermediate feature together with the original features of the two branches into the feature fusion module to obtain a new intermediate feature, and proceed to step 5. In step 4, the intermediate features, along with the features from the original two branches, are input into the feature fusion module to obtain new intermediate features, achieving the goal of deep fusion of the two modal features. The specific steps are as follows: Step 4-1: Extract the visible light image features from Step 3. Intermediate features after transposition Perform matrix multiplication, then ReLU activation, and finally combine with visible light image features. Perform matrix multiplication, then combine with intermediate features Performing bitwise addition yields... ; Step 4-2: Extract the infrared image features from Step 3. The features are input into the Self Attention network to obtain the features. ; Features Features after transposition Perform matrix multiplication, then ReLU activation, and then combine with the features again. Perform matrix multiplication, then combine with intermediate features Perform bitwise addition to obtain the feature. ; Step 4-3: Features The input is fed into the Self Attention network to obtain the output features. ; Step 4-4: Output features Perform BN normalization; Step 5: Input the image features and intermediate features of the two branches obtained in Step 3 and Step 4 into the loss function, including ID loss, trimodal triplet loss function and center bias loss function, so as to train a network with good performance and proceed to Step 6; Step 6: If the specified number of training rounds is reached, proceed to Step 7; otherwise, continue training and return to Step 1. Step 7: End.

2. The cross-modal pedestrian re-identification method based on intermediate features as described in claim 1, characterized in that, The steps for extracting preliminary image features in step 2 are as follows: Step 2-1: Input the visible light image into the feature extraction network to extract its preliminary features; Step 2-2: Input the infrared image into the feature extraction network to extract its preliminary features; Steps 2-3: Output features.

3. The cross-modal pedestrian re-identification method based on intermediate features according to claim 2, characterized in that, In step 2, the first layer of the residual network is used for feature extraction to fully perceive the shallow information of the image, and the feature extraction network parameters of the visible light image and the infrared image are not shared.

4. The cross-modal person re-identification method based on intermediate features as described in claim 3, characterized in that, The steps for extracting image features in step 3 are as follows: Step 3-1: Input the features extracted in Step 2 into a network consisting of the last four convolutional layers of ResNet50 and attention blocks of the feature mining module to obtain information-rich image features; Step 3-2: Perform MAX-pooling on the features from Step 3-1, then proceed to Step 3-3; Step 3-3: Output the pooling features of the image.

5. The cross-modal person re-identification method based on intermediate features according to claim 4, characterized in that, In step 3, the feature mining module introduces spatial attention and dimensional attention modules. The spatial attention module focuses on the dependency between local features and spatial location to enhance the perception of the target region; while the dimensional attention module focuses on channel information, adjusting the importance of different feature channels, enabling the network to dynamically allocate weights and improve the expressive power of key features. Input features The inputs are processed into two different attention modules, utilizing spatial attention and dimensional attention to obtain attention features that emphasize different information. These features are then multiplied with the input features, followed by activation and batch normalization. Finally, the features from both branches are multiplied again to obtain a feature that fuses spatial and dimensional attention. ; For the obtained features Global max pooling and global average pooling are performed respectively, and the features are concatenated to obtain a 2-dimensional feature vector. To further enhance the model's discriminative ability, matrix multiplication is then performed with the features that fuse spatial and dimensional attention, allowing the output features to fully integrate information from different attention mechanisms. The final result is a representation feature with the same dimensionality as the original input features but richer in detail. ; Finally, during deep network training, the vanishing and exploding gradient problems can lead to unstable model convergence. To avoid gradient descent, a residual connection mechanism is introduced in the final feature generation process, i.e. Add the original input features element by element.

6. The cross-modal pedestrian re-identification method based on intermediate features according to claim 5, characterized in that, In step 5, the id loss function is used to jointly constrain the model. The specific steps are as follows: Step 5-1: Pool the features extracted in Step 3; Step 5-2: Perform Batch Normalization on the features extracted in Step 5-1; Step 5-3: Pass the normalized features from Step 5-2 through a fully connected layer to reduce the dimensionality of the information while obtaining the dimensional information corresponding to the category, and proceed to Step 5-4; Step 5-4: Then input the features into the id loss function; Step 5-5: Using the losses from 4-4 and 5-4, calculate the gradient, perform backpropagation, and update the model parameters; During training, the ID loss is selected from the training set. Each identity is randomly selected. Zhang Xingren's images, each batch contains... Zhang pedestrian images; the identity loss for the pedestrian re-identification task is shown in the following formula: (1) in, Representing the Is Zhang's image identity...? , This represents the total number of pedestrian categories in the dataset. Indicates the first Zhang's image is for The probability of.

7. A cross-modal pedestrian re-identification method based on intermediate features as described in claim 6, characterized in that, In step 5, the trimodal triplet loss function uses the image features of the intermediate modality as anchor points to bring the farthest images of the same identity closer together and push away the closest images of different identities for the two original modal images respectively. For the extracted trimodal data, it is necessary to make full use of the generated intermediate modal features to achieve more efficient pedestrian identification. To this end, a trimodal triplet loss is proposed. The common hard triplet loss is a loss function widely used in pedestrian re-identification tasks. It brings the farthest positive pairs closer and pushes the nearest negative pairs further away. Simply put, it brings the distance between pedestrian image features taken by different cameras for the same type of person closer and pushes the distance between pedestrian image features for different people further away. The existing difficult cross-modal triplet loss, i.e., the difficult two-modal triplet loss; The same shape represents images of the same category, i.e., different modalities of the same pedestrian's identity; different shapes represent images of different categories, i.e., different pedestrians' identities. For each image in each batch, the distance to it is calculated, and the positive sample farthest from the image and the negative sample closest to the image are selected to calculate the loss, as shown in Equation (2): （2） In the formula, A is the anchor point sample. For samples with the same identity category as the anchor sample, For samples whose identity category differs from that of the anchor sample, For the threshold parameter, use Calculate the Euclidean distance to represent the characteristics. and The Euclidean distance is calculated using the formula shown in equation (3): （3） The trimodal triplet loss function uses the image features of the intermediate modality as anchor points, and pulls the farthest image of the same identity closer and pushes the nearest image of different identities further away for the two original modal images respectively. For each intermediate feature sample in each batch, it is used as an anchor sample, and the loss is calculated by comparing it with the positive sample farthest from the original modality image and the negative sample closest to the original modality image, as shown in Equation (4): （4） In the formula, A is the anchor point sample. As a positive sample, For negative samples, This is the threshold parameter.

8. The cross-modal pedestrian re-identification method based on intermediate features as described in claim 7, characterized in that, The center deviation loss function in step 5 takes into account the strong expressive power of visible light images and the fact that the ResNet50 network parameters are pre-trained with visible light images, which will lead to a strong visible light image bias in the model. In order to avoid the intermediate mode being affected by this factor, the center deviation loss function will be constrained for the feature center design of the three modes. First, for each type of pedestrian in the three modalities, the feature center is calculated, and the intermediate feature is constrained to be at the center of the two original features. The intermediate feature is also constrained to be equidistant from the two original features. For each unique ID image in each batch, the feature center for that category is calculated separately, and then the visible light features are obtained. Infrared characteristics The feature center is determined, and the distance between it and the intermediate feature center is calculated using the formulas shown in equations (5) and (6): （5） （6） In the formula, and These represent the visible light and infrared characteristics of the batch, respectively. This represents the number of images for each modality in each sample within the batch. This represents the number of sample categories in this batch. The center of the i-th intermediate feature, It is the center representing the visible light feature center and the infrared feature center of the i-th category; The total network loss is expressed as shown in formula (7): (7) in, and This is a hyperparameter, set separately in the experiment.

Images