A cross-modal person re-identification method based on cross-dimensional interactive attention

Abstract

The present invention discloses a cross-modal pedestrian re-identification method based on cross-dimensional interactive attention, comprising graying P*K visible light images to obtain grayscale images, inputting the obtained P*K grayscale images and P*K infrared images into a feature extraction network, using a ResNet50 network and an attention block based on cross-dimensional interaction to generate information-rich image features according to the input images, training a network with good performance through a group loss function and an id loss function, terminating the training if a specified number of training rounds is reached, otherwise continuing the training, and utilizing a cross-dimensional interactive attention mechanism to explore information interaction between dimensions, thereby significantly improving the recognition accuracy of the model.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision technology, and in particular relates to a cross-modal pedestrian re-identification method based on cross-dimensional interactive attention. Background Art

[0002] With the rapid development of social economy, public safety is becoming more and more important. At present, video surveillance equipment has spread to every corner of daily life, but the use of surveillance information is still in a relatively primitive stage. When an emergency occurs, it is very time-consuming and laborious to manually view surveillance videos to identify and retrieve; and in poor shooting environments, the accuracy of using the naked eye to distinguish is not high. Therefore, the industry has begun to study pedestrian re-identification (Person Re-identification, Re-ID) technology, which is one of the key technologies in intelligent security.

[0003] Pedestrian re-identification is to identify specific pedestrians from a large number of pedestrian images taken by cameras to meet the needs of finding people in various application scenarios. Due to its powerful functions, it is widely used in real life, such as danger warning, unmanned supermarkets, lost rescue and other quite broad application scenarios.

[0004] Early pedestrian re-identification was only for visible light pedestrian images. However, in actual applications, relying solely on single-modal pedestrian re-identification technology is far from meeting the needs of various application scenarios. Under poor lighting conditions, visible light cameras cannot obtain clear pedestrian images. If single-modal pedestrian re-identification technology is still used, the recognition accuracy will drop sharply. For this reason, monitoring equipment has been upgraded. The new monitoring equipment is equipped with cameras in both visible light and infrared modes, which can adaptively switch shooting modes according to lighting conditions to overcome the limitations of lighting conditions.

[0005] However, this also introduces a new problem, namely the need to process images of two modalities at the same time, which greatly increases the difficulty of the original single-modal pedestrian re-identification task. Therefore, in order to enable pedestrian re-identification to be used in many scenarios, the academic community has begun to study the "visible light-infrared" cross-modal pedestrian re-identification technology (Visible-Infrared Person Re-Identification, VI-ReID).

[0006] Figure 1The figure shows some pedestrian images in the dataset SYSU-MM01 proposed in the 2017 paper "RGB-infrared cross-modality person re-identification". The SYSU-MM01 dataset is the first dataset dedicated to cross-modal pedestrian re-identification tasks. Cameras 1 and 2 take indoor visible light pedestrian images, cameras 4 and 5 take outdoor visible light pedestrian images, and cameras 3 and 6 take infrared images of pedestrians. The training set of SYSU-MM01 has a total of 395 pedestrian identities, including 22,258 visible light images and 11,909 infrared images. The test set consists of images of another 96 pedestrians and is divided into two evaluation modes, namely All-Search mode and Indoor-Search mode.

[0007] Figure 2 The figure shows some pedestrian images in the dataset RegDB proposed in the 2017 paper "Unlabeled Samples Generated by GAN Improve the Person Re-identification Baseline in Vitro". The RegDB dataset has poor clarity, small changes in pedestrian postures, and is quite different from actual application scenarios, so the recognition difficulty is relatively low. The dataset contains a total of 412 pedestrians, each with 10 visible light images and 10 infrared images. The experiment randomly selects 2060 visible light images and 2060 near-infrared images corresponding to 206 pedestrians for training, and the remaining 206 pedestrian images are used for testing. In addition, two retrieval modes are usually set: visible light image retrieval infrared image (Visible-to-Thermal) and infrared image retrieval visible light image (Thermal-to-Visible).

[0008] From the diagram, we can see that even for the same person, there are significant differences due to being in different modes, different shooting angles, different pedestrian postures, different background environments, and occlusion.

[0009] In recent years, many cross-modal pedestrian re-identification methods have been proposed. For example, paper 3 "Visible Thermal Person Re-Identification via Dual-Constrained Top-Ranking" proposed a two-way dual-constrained loss dual-stream network (BDTR), designed a dual-constrained ranking loss to alleviate cross-modal differences, and used identity loss to reduce the intra-class distance to achieve the purpose of simultaneously processing intra-modal and cross-modal differences. Although the model takes into account the intra-modal and cross-modal differences, the model still does not pay attention to the information interaction between dimensions and still uses the most common triplet loss function. Its model structure is as follows Figure 3 shown.

[0010] In order to address the current technical shortcoming of not paying attention to the interaction of information between dimensions, the present invention proposes a cross-modal person re-identification method based on cross-dimensional interactive attention. Summary of the invention

[0011] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a cross-modal pedestrian re-identification method based on cross-dimensional interactive attention, which utilizes the cross-dimensional interactive attention mechanism to explore the information interaction between dimensions and greatly improves the recognition accuracy of the model.

[0012] Technical solution: To achieve the above purpose, the technical solution of the present invention is as follows:

[0013] A cross-modal person re-identification method based on cross-dimensional interactive attention includes the following steps:

[0014] Step 1: grayscale the P*K visible light images to obtain grayscale images, and then proceed to step 2;

[0015] Step 2: Input the P*K grayscale images and P*K infrared images obtained in step 1 into the feature extraction network and proceed to step 3;

[0016] Step 3: Use the ResNet50 network and the attention block based on cross-dimensional interaction to generate informative image features based on the image input in step 2, and proceed to step 4;

[0017] Step 4: Train a network with good performance through the group loss function and the id loss function, and proceed to step 5;

[0018] Step 5: If the specified number of training rounds is reached, proceed to step 6, otherwise continue training and return to step 1;

[0019] Step 6: End.

[0020] Furthermore, the step 2 comprises the following steps:

[0021] Step 2-1: Input the grayscale image into the feature extraction network to extract its preliminary features;

[0022] Step 2-2: Input the infrared image into the feature extraction network to extract its preliminary features;

[0023] Step 2-3: Output features.

[0024] Further, the step 3 comprises the following steps:

[0025] Step 3-1: Input the features obtained in step 2 into the network composed of the convolutional layers of the last four layers of ResNet50 and the cross-dimensional interactive attention block to obtain information-rich image features;

[0026] Step 3-2: Pass the image features obtained in step 3-1 through the M-pooling layer and enter step 3-3;

[0027] Step 3-3: Output the pooled features of the image.

[0028] Furthermore, in step 3, the attention block based on cross-dimensional interaction is embedded in the ResNet50 network to fully perceive the information between different dimensions, enhance the connection between feature dimensions, and mine the information between dimensions.

[0029] Furthermore, in step 4, the features are pooled using M-pooling, where the M-pooling module is a combination of four types of operations: average pooling, maximum pooling, averaging, and maximum value, to obtain and respectively, and concatenate them according to the last dimension to obtain.

[0030] Furthermore, in step 4, the group loss function and the id loss function are used to constrain the model. The specific steps are as follows:

[0031] Step 4-1: Input the pooled features extracted in step 3 into the group loss function;

[0032] Step 4-2: Normalize the features extracted in step 3 using the BN layer;

[0033] Step 4-3: Pass the normalized features in step 4-2 through the fully connected layer to reduce the dimension of the information while obtaining the dimensional information corresponding to the category, and then proceed to step 4-4;

[0034] Step 4-4: Input the features into the id loss function;

[0035] Step 4-5: Using the losses of 4-1 and 4-4, calculate the gradient, back propagate, and update the model parameters.

[0036] Furthermore, the distance between samples in the extracted pedestrian features is calculated, and different groups represent different pedestrian identities. The sample distance calculation is converted into first calculating the mean and then the variance. The intra-group loss can be expressed as formula (3):

[0037]

[0038] Among them: m represents the mean point of the intra-group feature, and the final calculated intra-group distance is also recorded as var.

[0039] Furthermore, while expanding the distance between groups, the variance within each group is constrained to remain unchanged, and the inter-group loss is formula (4):

[0040]

[0041] Where: S i and S j represents two pedestrian identity groups, and They represent a sample feature in each group, var i and var j This is the proposed constraint, which represents the intra-group distance between the two groups;

[0042] Therefore, the total group loss can be expressed as formula (5):

[0043] L group =L intra +L inter Formula (5).

[0044] Furthermore, when the ID loss is used in training, P identities are selected in the training set, and K pedestrian images are randomly selected from each identity. There are 2P*K pedestrian images in each Batch. The identity loss for the pedestrian re-identification task is as follows:

[0045]

[0046] Among them, y i,k represents whether the identity of the i-th image is k, N represents the total number of pedestrian categories in the dataset, and p i,k Indicates the probability that the identity of the i-th image is k.

[0047] Furthermore, through the network total loss function L overall By restricting the network, the final network total loss function L overall As shown in formula (7):

[0048] L overall =Lid +λL group Formula (7);

[0049] Among them, λ is a hyperparameter.

[0050] Beneficial effect: The present invention designs a network with ResNet50 as the backbone network and embeds a cross-dimensional interactive attention module, which, combined with the proposed group loss, improves the recognition accuracy of the model.

[0051] First, the present invention proposes to integrate a residual network based on a cross-dimensional interactive attention mechanism, and adds a cross-dimensional interactive attention module to fully perceive the information between different dimensions, enhance the connection between feature dimensions, and mine richer information between dimensions.

[0052] Secondly, the present invention proposes a group loss function and combines it with the id loss function to jointly train the network, which effectively improves the recognition accuracy of the model.

[0053] In the SYSU-MM01 and RegDB public datasets, the correct recognition rate (Rank-1) and mean average precision (mAP) of the first recognition result of the method of the present invention are 54.43%, 68.96% and 72.59%, 69.1%, respectively, which is higher than other models. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Attached Figure 1 Pedestrian images in color and infrared modes in the SYSU-MM01 dataset;

[0055] Attached Figure 2 These are pedestrian images in color and infrared modes in the RegDB dataset;

[0056] Attached Figure 3 It is the network model proposed in the prior art paper 3;

[0057] Attached Figure 4 Schematic diagram of the network framework of person re-identification based on the cross-dimensional attention mechanism of the present invention;

[0058] Attached Figure 5 This is a flow chart of the training phase of the present invention;

[0059] Attached Figure 6 The structural diagram of the cross-dimensional interactive attention module is used to learn the information between dimensions;

[0060] Attached Figure 7 Schematic diagram of the cross-modal person re-identification task. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with the accompanying drawings.

[0062] As attached Figure 4 The figure shows a schematic diagram of a pedestrian re-identification network framework based on a cross-dimensional attention mechanism of the present invention, and discloses a cross-modal pedestrian re-identification method based on cross-dimensional interactive attention. The training flow chart of the network proposed in the present invention is as follows: Figure 5 As shown in the figure. The training process adopts mini-batch training. P pedestrians are randomly selected in each batch, and K visible light and infrared images are randomly selected for these pedestrians. This program will take an image as an example to introduce the training process, which includes the following steps:

[0063] Step 1: grayscale the P*K visible light images to obtain grayscale images, and then proceed to step 2;

[0064] Step 2: Input the P*K grayscale images and P*K infrared images obtained in step 1 into the feature extraction network and proceed to step 3;

[0065] Step 3: A network with ResNet50 as the backbone network is used to generate informative image features based on the image input in step 2 using the ResNet50 network and the attention block based on cross-dimensional interaction, and then proceed to step 4;

[0066] Step 4: Train a network with good performance through the group loss function and the id loss function, and proceed to step 5;

[0067] Step 5: If the specified number of training rounds is reached, proceed to step 6, otherwise continue training and return to step 1;

[0068] Step 6: End.

[0069] Preferably, the images in step 1 are all from standard datasets for pedestrian re-identification, such as SYSU-MM01 and RegDB datasets. Visible light images are composed of three primary colors: red, green, and blue, and contain three channels, each corresponding to a primary color. Infrared images have only one channel. The present invention assumes that there are 2P*K pedestrian images in total, and the image samples can be expressed as Among them, I i represents the i-th pedestrian image, y i Indicates I i The present invention uses 2P*K samples {I i ,y i} as an example to introduce the working principle of the present invention during the training process.

[0070] In step 1, the visible light image is grayscaled to alleviate cross-modal differences, and the subsequent network is trained only using the processed grayscale image and infrared image.

[0071] The step 2 comprises the following steps:

[0072] Step 2-1: Input the grayscale image into the feature extraction network to extract its preliminary features;

[0073] Step 2-2: Input the infrared image into the feature extraction network to extract its preliminary features;

[0074] Step 2-3: Output features.

[0075] In step 2, the present invention uses the first layer of the residual network to perform feature extraction, fully perceives the shallow information of the image, and the feature extraction network parameters of the grayscale image and the infrared image are not shared.

[0076] The input images in steps 1 and 2 are subjected to three data augmentation methods: random cropping, horizontal flipping, and erasing to enhance the robustness of the model.

[0077] The step 3 comprises the following steps:

[0078] Step 3-1: Input the features obtained in step 2 into the network composed of the convolutional layers of the last four layers of ResNet50 and the cross-dimensional interactive attention block to obtain information-rich image features;

[0079] Step 3-2: Pass the image features obtained in step 3-1 through the M-pooling layer and enter step 3-3;

[0080] Step 3-3: Output the pooled features of the image.

[0081] In step 3, the present invention uses a residual network and a cross-dimensional interactive attention mechanism to extract features. Among them, the ResNet50 network is a typical network in the residual network, which includes 50 two-dimensional convolution operations. The present invention uses the ResNet50 network to increase the network depth and improve the feature expression ability. Unlike the feature extraction network in step 2, the network parameters are shared.

[0082] In step 3, the attention block based on cross-dimensional interaction is embedded in the ResNet50 network to fully perceive the information between different dimensions, enhance the connection between feature dimensions, and mine richer information between dimensions.

[0083] The core idea of ​​cross-dimensional interactive attention is to learn the connections between the channel dimension (C) and the height (H) and width (W) dimensions in the spatial dimension respectively.

[0084] Assume that the input feature of the cross-dimensional interactive attention module is denoted by F∈R C×H×W , where C, H, and W represent the three dimensions of channel, height, and width respectively. First, the input tensor is dimensionally permuted according to two strategies and then input into the M-pooling module. Its essence is to pool H and W in the spatial dimension respectively. The M-pooling module is a combination of four types of operations: average pooling, maximum pooling, averaging, and maximum value. F is obtained respectively. H ∈R 4×C×W and F W ∈R 4×C×H , and concatenate according to the last dimension to get F Pooling ∈R 4×C×(H+W) .

[0085] Next, the output F of the previous stage Pooling , the Unfold function is used to convert the three-dimensional features into two dimensions. This operation only involves convolution but not convolution, and the channel and spatial dimensions are fused. Since the fused dimension is high, this paper uses Conv1d convolution and selects a larger one-dimensional convolution kernel to learn the interactive information of the channel and spatial dimensions (H and W dimensions). Finally, the fold function is used to restore it and obtain the feature F after the dimension interaction. Interaction ∈R 1×C×(H+W) .

[0086] Finally, the attention part is similar to the conventional attention mechanism. This paper uses the Sigmoid function as the activation function. Through segmentation and dimension permutation operations, the feature attention map M in the C and H dimensions under the initial two strategies is obtained. CH ∈R C×1×W And the feature attention map M in C and W dimensions CW ∈R C×H×1 Finally, the broadcast mechanism is used to obtain the final output, which is expressed as shown in formula (1).

[0087] Output=F·(M CH +M CW ) Formula (1)

[0088] Among them, the structure of the attention block is as follows Figure 6 shown.

[0089] In the step 4, the features are pooled using M-pooling, where the M-pooling module is a combination of four types of operations: average pooling, maximum pooling, averaging, and maximum value, and respectively obtains and, and concatenates them according to the last dimension.

[0090] In step 4, the group loss function and the id loss function are used to constrain the model. The specific steps are as follows:

[0091] Step 4-1: Input the pooled features extracted in step 3 into the group loss function;

[0092] Step 4-2: Normalize the features extracted in step 3 using the BN layer;

[0093] Step 4-3: Pass the normalized features in step 4-2 through the fully connected layer to reduce the dimension of the information while obtaining the dimensional information corresponding to the category, and then proceed to step 4-4;

[0094] Step 4-4: Input the features into the id loss function;

[0095] Step 4-5: Using the losses of 4-1 and 4-4, calculate the gradient, back propagate, and update the model parameters.

[0096] The present invention proposes a group loss function to solve the problem that uneven data distribution may cause the model training process to be unstable and converge slowly. The purpose of metric learning is to reduce the intra-class distance as much as possible and expand the inter-class distance. Common methods usually combine identity loss with triplet loss for metric learning, but triplet loss requires the selection of triplets, which easily leads to uneven data distribution and may cause the model training process to be unstable and converge slowly. Therefore, in order to improve the triplet loss function, this scheme designs a group loss loss function to replace the triplet loss function, and combines the identity loss to constrain the overall network. Its schematic diagram is shown as follows: Figure 7 As shown in the figure, the same color represents the same pedestrian identity, and the solid and hollow represent two modes respectively.

[0097] Group loss is divided into two parts: intra-group and inter-group. First, the distance between samples in the group is calculated for the extracted pedestrian features. Different groups represent different pedestrian identities. Assuming that each batch selects P pedestrian identities in the training set, and randomly selects K visible light images and K infrared images for each pedestrian identity, the number of samples in the group is 2K, denoted as N. The intra-group loss can be expressed as shown in formula (2).

[0098]

[0099] where f s , f t represents the characteristics of two samples in the group. In this form, the time complexity of the intra-group loss is relatively high. This paper further simplifies it by converting the sample pair distance calculation into first calculating the mean and then the variance. The intra-group loss can be expressed as formula (3):

[0100]

[0101] Among them: m represents the mean point of the intra-group feature, and the final calculated intra-group distance is also recorded as var.

[0102] For the inter-group, it is necessary to expand the distance between the samples as much as possible. In order to prevent the previously shortened samples within the group from being dispersed when the distance between the samples between the groups is expanded, the intra-group variance of each group is constrained to remain unchanged while expanding the distance between the groups. The inter-group loss is formula (4):

[0103]

[0104] Where: S i and S j represents two pedestrian identity groups, and They represent a sample feature in each group, var i and var j This is the proposed constraint, which represents the intra-group distance between the two groups;

[0105] Therefore, the total group loss can be expressed as formula (5):

[0106] L group =L intra +L inter Formula (5).

[0107] The id loss function described in this paper is similar to other loss functions. When the id loss is trained, P identities are selected in the training set, and K pedestrian images are randomly selected from each identity. There are 2P*K pedestrian images in each batch. The identity loss for the pedestrian re-identification task is as shown in formula (6):

[0108]

[0109] Among them, y i,k represents whether the identity of the i-th image is k, N represents the total number of pedestrian categories in the dataset, and p i,k Indicates the probability that the identity of the i-th image is k.

[0110] Furthermore, through the network total loss function L overall By restricting the network, the final network total loss function L overall As shown in formula (7):

[0111] L overall =L id +λL group Formula (7);

[0112] Among them, λ is a hyperparameter, which is a preset constant and is set separately in the experiment.

[0113] Existing pedestrian re-identification methods ignore the information interaction between different dimensions, which is not conducive to mining rich pedestrian information, which greatly affects the accuracy of pedestrian re-identification. To this end, the present invention proposes a cross-modal pedestrian re-identification method based on cross-dimensional interactive attention, thereby improving the recognition accuracy.

[0114] The test process of the present invention is:

[0115] Step 1: Enter the query set and gallery set, and proceed to step 2;

[0116] Step 2: Using the model obtained during the training process, perform feature extraction on all pedestrian images in the query set and gallery set input in step 1, and proceed to step 3;

[0117] Step 3: Calculate the similarity between the query set features and the gallery set features, and proceed to step 4;

[0118] Step 4: According to the similarity, obtain the matching result corresponding to each pedestrian image in the query set and proceed to step 5;

[0119] Step 5: End.

[0120] Preferably, the query set in step 1 of the test process represents a set of pedestrian images to be queried, and the gallery set represents a set of pedestrian images to be matched by the query set.

[0121] Preferably, the similarity calculation method in step 3 of the test process is point product similarity.

[0122] Preferably, in step 4 of the test process, each image in the query set has several images matched from the gallery set. The evaluation indicators are the Cumulative Matching Characteristic (CMC) and Mean Average Precision (mAP), which are commonly used in cross-modal person re-identification tasks. Rank-n represents the probability of the correct identity appearing in the first n items of the retrieval sequence, and mAP reflects the overall performance of the method when there are multiple targets.

[0123] This embodiment will use the mini-batch gradient descent method to update the model parameters for the model of the present invention, that is, each time the gradient descends, a small batch of samples is randomly selected for parameter update. The ResNet50 network is selected as the basic skeleton, and the parameters of ResNet50 are pre-trained on ImageNet, and the stride of the last bottleneck block of the teacher network is set to 1. In terms of image preprocessing, three data enhancement methods, random cropping, horizontal flipping and erasing, are considered. In the experiment, the image size is uniformly set to 256×128. The momentum parameter of this experiment is set to 0.9, the initial learning rate is set to 0.01, the warm-up learning rate strategy is used in the first 10 epochs, and the 20th and 50th epochs are 0.01 and 0.001 respectively, for a total of 80 rounds of training. During testing, pedestrian features that have passed through the BN layer are used for retrieval, and cosine distance is used to measure the distance between pedestrians. This experiment uses the pytorch1.8 deep learning framework and uses an NVIDIA3090 for acceleration.

[0124] Embodiment 1:

[0125] This example will use the SYSU-MM01 dataset and the RegDB dataset to complete the person re-identification task and test the performance of the model to evaluate the method in this paper. The performance comparison results of each method are shown in Table 1 and Table 2.

[0126] Table 1 Performance comparison of the proposed method and other methods on the SYSU-MM01 dataset

[0127]

[0128] Table 2 Performance comparison of the method of the present invention and other methods on the RegDB dataset

[0129]

[0130]

[0131] First, we focus on the SYSU-MM01 dataset. Methods such as BDTR only consider the metric learning part, but ignore the improvement of the feature extraction stage. Experimental results show that compared with the proposed method, in the full search mode, Rank-1 and mAP are improved by 27.11% and 25.14% respectively; in the indoor search mode, Rank-1 and mAP are improved by 30.81% and 27.10% respectively. It can be seen that the proposed method designs cross-dimensional interactive attention to integrate feature extraction, optimizes the feature extraction stage, enhances feature representation, and significantly improves network performance.

[0132] Methods such as D2RL and AlignGAN use GAN to alleviate cross-modal differences, but the generated images are prone to produce additional noise, which can affect the training effect of the model. This paper grayscales the visible light modality images and then performs feature alignment to achieve the effect of processing cross-modal differences. Compared with the optimal method AlignGAN using GAN, in the full search mode, Rank-1 and mAP are improved by 12.03% and 11.76% respectively; in the indoor search mode, Rank-1 and mAP are improved by 16.83% and 14.66% respectively.

[0133] In the above comparative experiments, except for the method in this paper, the AGW method has the best effect. The AGW method adds a non-local attention mechanism in the feature extraction stage to make the extracted pedestrian features more global, and designs a weighted regularized triple loss combined with identity loss for metric learning, which achieves good results. On this basis, the method in this paper improves Rank-1 and mAP by 6.93% and 4.81% respectively in the full search mode; in the indoor search mode, Rank-1 and mAP improve by 8.56% and 5.99% respectively. This shows that the combination of cross-dimensional interactive attention and group loss designed in this paper has a better effect.

[0134] At the same time, for the RegDB dataset, the evaluation indicators of this paper are also higher than the existing methods. Compared with the second-best method AGW, in the visible-to-thermal mode, Rank-1 and mAP are improved by 2.54% and 2.10% respectively; in the thermal-to-visible mode, Rank-1 and mAP are improved by 3.44% and 3.20% respectively.

[0135] Embodiment 2:

[0136] This embodiment will introduce an applicable scenario of the present invention.

[0137] Cameras are installed in many public places, such as shopping malls, busy streets, and major traffic arteries. Security personnel can call up nearby cameras and identify the identity of the relevant target person from the most recent video, thereby obtaining the target person's trajectory and finally finding the target person.

[0138] First, the security personnel obtain the original video from the camera in the area, and use the relevant technology of pedestrian detection to crop the pedestrian images that appear in the surveillance video, and use the camera name and shooting time as the image name.

[0139] Next, the night picture of the target person and the cropped pedestrian image are input into the model.

[0140] Then, the model is used to find pictures where the target person has appeared, and the target person's movement trajectory is analyzed using the picture names.

[0141] Finally, use the trajectory route to find the target person.

[0142] The model only needs to input infrared images at night or visible light images during the day, and combined with nearby cameras, it can automatically find out the place and time where the target person appears. Security personnel can then analyze the movement trajectory of the target person, thereby finding the target person faster and improving security search efficiency.

[0143] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A cross-modal person re-identification method based on cross-dimensional interactive attention, characterized by: The following steps are involved: Step 1: grayscale the P*K visible light images to obtain grayscale images, and then proceed to step 2; Step 2: Input the P*K grayscale images and P*K infrared images obtained in step 1 into the feature extraction network and proceed to step 3; The step 2 comprises the following steps: Step 2-1: Input the grayscale image into the feature extraction network to extract its preliminary features; Step 2-2: Input the infrared image into the feature extraction network to extract its preliminary features; Step 2-3: Output features; Step 3: Use the ResNet50 network and the attention block based on cross-dimensional interaction to generate informative image features based on the image input in step 2, and proceed to step 4; The step 3 comprises the following steps: Step 3-1: Input the features obtained in step 2 into a network consisting of the last four convolutional layers of ResNet50 and a cross-dimensional interactive attention block to obtain information-rich image features; Step 3-2: Pass the image features obtained in step 3-1 through the M-pooling layer and enter step 3-3; Step 3-3: Output the pooled features of the image; In step 3, the attention block based on cross-dimensional interaction is embedded in the ResNet50 network to fully perceive the information between different dimensions, enhance the connection between feature dimensions, and mine the information between dimensions; M-pooling is used to pool the features, where the M-pooling module is a combination of four operations: average pooling, maximum pooling, averaging, and maximum value. Assume that the input feature of the cross-dimensional interactive attention module is denoted by F∈R C×H×W , where C, H, and W represent the three dimensions of channel, height, and width respectively; first, the input tensor is dimensionally permuted according to two strategies and then input into the M-pooling module, which essentially pools H and W in the spatial dimension respectively; the M-pooling module is a combination of four types of operations: average pooling, maximum pooling, averaging, and maximum value, and obtains F respectively. H ∈R 4×C×W and F W ∈R 4×C×H , and concatenate according to the last dimension to get F Pooling ∈R 4×C×(H+W) ; Next, the output F of the previous stage Pooling , use the Unfold function to convert the three-dimensional features into two dimensions, and this operation is only convolution, which fuses the channel and spatial dimensions; then use the Conv1d convolution to select a one-dimensional convolution kernel to learn the interactive information of the channel and spatial dimensions H and W, and finally use the fold function to restore it to obtain the feature F after dimensional interaction Interaction ∈R 1×C×(H+W) ; Finally, the attention part uses the Sigmoid function as the activation function. Through segmentation and dimension permutation operations, the feature attention map M in the C and H dimensions under the first two strategies is obtained. CH ∈R C×1×W And the feature attention map M in C and W dimensions CW ∈R C ×H×1 ; The final output is obtained by using the broadcast mechanism, and its expression is shown in formula (1); Output=F·(M CH +M CW ) Formula (1) Step 4: Train a network with good performance through the group loss function and the id loss function, and proceed to step 5; Step 5: If the specified number of training rounds is reached, proceed to step 6, otherwise continue training and return to step 1; Step 6: End.

2. The cross-modal person re-identification method based on cross-dimensional interactive attention according to claim 1, characterized in that: In step 4, the group loss function and the id loss function are used to constrain the model. The specific steps are as follows: Step 4-1: Input the pooled features extracted in step 3 into the group loss function; Step 4-2: Normalize the features extracted in step 3 using the BN layer; Step 4-3: Pass the normalized features in step 4-2 through the fully connected layer to reduce the dimension of the information and obtain the dimensional information corresponding to the category, and then proceed to step 4-4; Step 4-4: Input the features into the id loss function; Step 4-5: Using the losses of 4-1 and 4-4, calculate the gradient, back propagate, and update the model parameters.

3. The cross-modal person re-identification method based on cross-dimensional interactive attention according to claim 1, characterized in that: The distance between samples in the extracted pedestrian features is calculated. Different groups represent different pedestrian identities. The distance calculation of the sample pairs is converted into first calculating the mean and then the variance. The loss within the group can be expressed as formula (3): Among them: m represents the mean point of the intra-group feature, and the final calculated intra-group distance is also recorded as var.

4. The cross-modal person re-identification method based on cross-dimensional interactive attention according to claim 3 is characterized by: While expanding the distance between groups, constraining the variance within each group to remain unchanged, the inter-group loss is formula (4): Where: S i and S j represents two pedestrian identity groups, and They represent a sample feature in each group, var i and var j This is the proposed constraint, which represents the intra-group distance between the two groups; Therefore, the total group loss can be expressed as formula (5): L group =L intra +L inter Formula (5).

5. The cross-modal person re-identification method based on cross-dimensional interactive attention according to claim 4, characterized in that: When the ID loss is trained, P identities are selected in the training set, and K pedestrian images are randomly selected from each identity. There are 2P*K pedestrian images in each Batch. The identity loss for the pedestrian re-identification task is as shown in formula (5): Among them, y i,k represents whether the identity of the i-th image is k, N represents the total number of pedestrian categories in the dataset, and p i,k Indicates the probability that the identity of the i-th image is k.

6. The cross-modal person re-identification method based on cross-dimensional interactive attention according to claim 4, characterized in that: Through the network total loss function L overall By restricting the network, the final network total loss function L overall As shown in formula (6): L overall =L id +λL group Formula (7); Among them, λ is a hyperparameter.

Images