A cross-modal person re-identification method using local supervision

Through the cross-modal pedestrian re-identification method with local supervision, shared features are extracted using Resnet50 and Nonlocal attention modules, and supervised in combination with local features to design cross-modal grouping losses, solving the problems of cross-modal differences and internal differences in modalities. , significantly improves the recognition rate and is better than existing methods.

CN115565204BActive Publication Date: 2024-12-03XINGCHEN XINRUI (SHENZHEN) TECHNOLOGY CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202211223245.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-08
Publication Date
2024-12-03
Estimated Expiration
2042-10-08

AI Technical Summary

Technical Problem

In the cross-modal pedestrian re-identification task, the existing methods are difficult to take into account both cross-modal differences and modal internal differences, resulting in low recognition rate. The GAN method introduces noise and requires additional training processes, which is not effective and only uses global features. insufficient robustness.

Method used

A cross-modal pedestrian re-identification method using local supervision is proposed. Shared features are extracted through Resnet50 and Nonlocal attention modules, and global features are supervised in combination with local features, cross-modal packet loss is designed, and joint identity loss is combined to constrain the network. Enhance the robustness to background, occlusion and other noises.

Benefits of technology

On the SYSU-MM01 and RegDB datasets, the recognition rate-related evaluation indicators Rank-1 and mAP reached 53.31%, 50.88%, 73.51%, and 68.55%, respectively, which was better than similar methods and significantly improved the cross-modal pedestrian re-identification performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115565204B_ABST
    Figure CN115565204B_ABST
Patent Text Reader

Abstract

The present invention discloses a cross-modal pedestrian re-identification method using local supervision. First, an image processing method is used to convert a visible light image into a grayscale image that is closer to an infrared image, thereby alleviating cross-modal differences at the image level and avoiding the problem of introducing new noise and requiring additional training processes that may be caused by using GAN. Secondly, a two-stream network with shared parameters is used to extract discriminative shared features to alleviate cross-modal differences at the feature level; then, a local supervision network is designed so that the learned global features absorb the advantages of local features, enhance their robustness to background, occlusion and other noise, and alleviate internal modal differences; finally, a cross-modal grouping loss is designed, and the network is constrained by the joint identity loss to maintain the intra-class distance while increasing the inter-class distance to improve the overall performance. The cross-modal pedestrian re-identification framework LSN of the present invention takes into account both cross-modal differences and intra-modal differences.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer vision technology, in particular to the field of cross-modal pedestrian re-identification, and specifically to a cross-modal pedestrian re-identification method using local supervision. Background Art

[0002] With the continuous development of artificial intelligence, smart city construction projects have received widespread attention at home and abroad in recent years. As an indispensable part of it, smart security has become a hot topic of current research. To do a good job in smart security, how to accurately identify the identity of pedestrians in daily life scenarios has become a key issue that needs to be solved urgently. For example, a theft occurred in a shopping mall. Based on the surveillance video at the scene of the theft, we obtained the image of the thief. We need to identify it in all the surveillance data of the entire shopping mall, and then confirm the identity of the thief and his / her path, and arrest him / her.

[0003] At present, face recognition technology is relatively mature, but it requires high-quality frontal images of pedestrians. However, surveillance cameras in most scenes of daily life cannot obtain images that meet the requirements and can only capture the overall image of pedestrians. As mentioned above, thieves often wear hats and masks to cover their facial features. In this non-cooperative scenario, person re-identification technology is proposed as a supplementary technology to face recognition to identify pedestrians.

[0004] Person re-identification refers to the technology of retrieving pedestrian images with the same identity from a large-scale pedestrian image library given a pedestrian image to be queried in a scene captured by multiple non-overlapping cameras. In early research, it focused more on visible light images captured by visible light cameras during the day, which are single-mode images. In practical applications, visible light cameras can only meet the needs of some scenes. At night or in low light conditions, visible light cameras cannot obtain clear feature information of pedestrians, which affects the accuracy of recognition. For this reason, the industry began to introduce infrared cameras to make up for this defect. In this way, high-quality visible light pedestrian images can be taken by visible light cameras during the day or under good lighting conditions, and infrared pedestrian images with less dependence on lighting can be taken by infrared cameras at night or in low light environments, which better get rid of the limitations of lighting conditions. However, processing images of two modes at the same time also increases the difficulty of pedestrian re-identification tasks, forming cross-modal pedestrian re-identification based on visible light and infrared (VI-ReID).

[0005] The cross-modal person re-identification task faces two major challenges. The first is the intra-modality difference. Since each pedestrian may be photographed by a different camera, these cameras have different viewpoint positions and shooting angles. The pedestrian images obtained will inevitably have different postures, different backgrounds, occlusions, etc. Figure 1 As shown. For example, in the above example, thieves usually choose places where people gather to commit crimes, so that they can use the cover of the surrounding crowds, resulting in the pedestrian images of thieves collected by the camera being seriously interfered by background, occlusion and other noise. In this way, even for the same pedestrian, the similarity between his own images may be very low, or even lower than the similarity between other pedestrian images, resulting in a situation where the intra-class difference is greater than the inter-class difference. The second is the cross-modal difference. Although we have introduced infrared cameras to get rid of the limitations of lighting conditions. However, due to the different wavelength ranges of visible light images and infrared images, and the number of channels of visible light images is 3, while the number of channels of infrared images is 1, there is a huge difference between visible light images and infrared images in essence, such as Figure 2 For example, in the above example, after committing theft, the thief usually chooses places such as underground parking lots with poor lighting conditions to hide or flee. In this case, we first obtain the visible light image of the thief taken by the visible light camera at the theft scene, and now the infrared image of the thief taken by the infrared camera at a place with poor lighting conditions. We need to identify pedestrian images in these two modes, which greatly increases the difficulty of identification.

[0006] The end-to-end aligned generative adversarial network (AlignGAN) proposed by Wang et al. uses CycleGAN to perform style transfer and generate pseudo infrared images to make up for the gap between visible light images and infrared images. However, it inevitably introduces additional noise and requires additional training process, which consumes a lot of time and affects the application efficiency in real scenarios.

[0007] The AGW method proposed by Ye et al. designed a baseline with a non-local attention mechanism, which obtains information from the middle and high layers and enhances the discriminability of features. However, the model extracts shared features based on global features, and the samples contain a lot of noise with interfering properties such as background and occlusion. The model that only uses global features has poor robustness to noise. Summary of the invention

[0008] Purpose of the invention: The cross-modal differences and intra-modal differences that are mainly faced in the cross-modal pedestrian re-identification task have been alleviated to varying degrees under the action of existing methods, but there are few methods that can take both into account well, resulting in the recognition rate of the cross-modal pedestrian re-identification task is still not high. The existing method uses GAN to achieve modal unification, which is easy to introduce new noise and requires an additional training process, and the effect is not good; only using global features cannot ensure robustness to interference such as occlusion. Starting from the shortcomings of the existing methods, the present invention proposes a cross-modal pedestrian re-identification method using local supervision to further improve the recognition rate of the cross-modal pedestrian re-identification task.

[0009] Technical solution: A cross-modal pedestrian re-identification method using local supervision. A cross-modal pedestrian re-identification framework LSN using local supervision is established. Assume that each time training, P pedestrian identities are randomly selected in the training set, and K visible light images and K infrared images are randomly selected for each identity. The details are as follows:

[0010] Step 1: Input K visible light pedestrian images and K infrared pedestrian images to the LSN model and proceed to step 2;

[0011] Step 2: If it is an infrared pedestrian image, go to step 3; if it is a visible light pedestrian image, convert it into a grayscale image using image processing methods and go to step 3;

[0012] Step 3: Combine Resnet50 and Nonlocal attention modules to extract global shared features of infrared images and grayscale images, and proceed to step 4;

[0013] Step 4: Use the shared features to extract global features and local features respectively, and then proceed to step 5;

[0014] Step 5: Use local features to supervise global features, use cross-modal grouping loss combined with identity loss to constrain global features, and update network parameters through back propagation, and go to step 6;

[0015] Step 6: If the specified number of training rounds is reached, go to step 7, otherwise return to step 1;

[0016] Step 7: Training is complete.

[0017] Furthermore, the pedestrian images in step 1 are all from the standard datasets SYSU-MM01 and RegDB for cross-modal pedestrian re-identification; each pedestrian image is randomly cropped and randomly horizontally flipped to enhance the generalization ability of the experiment, and unified into a size of 256×128; assuming represents the i-th visible light image, represents the jth infrared image, and Respectively and The corresponding pedestrian identity; among them, input the same pedestrian The visible light modal sample is The infrared modality samples are

[0018] Furthermore, in step 2, for the pedestrian image in the visible light modality, a grayscale image that is closer to the infrared image is selected as a transition. Before the visible light image is input into the network training, it is first converted into a grayscale image. After the conversion, the label information of the original sample is still retained, which is recorded as

[0019] Furthermore, the detailed process of step 3 is as follows:

[0020] Step 3-1: Pass the processed grayscale and infrared pedestrian images through a layer of convolutional network to extract the shallow-layer unique features of the two modes. and

[0021] Step 3-2: and The input is a network composed of the last four layers of Resnet50 and the Nonlocal attention module, and the shared features of the two modalities are extracted. and

[0022] Furthermore, in the step 3-1, the convolutional network structures of the two modes are the same but the parameters are different.

[0023] Furthermore, in the step 3-2, the shallow-layer specific features of the two modalities are input into a network block with the same structure and parameters, and the features of the two different modalities are mapped to the same feature subspace to achieve feature alignment, thereby alleviating cross-modal differences at the feature level; the network block consists of the last four layers of the residual network Resnet50 and the Nonlocal attention module.

[0024] Furthermore, the detailed process of extracting global features in step 4 is as follows:

[0025] Step 4-1: Perform adaptive average pooling on the shared features of the two modalities to obtain the preliminary global features and

[0026] Step 4-2: Input the prepared global features into the BN layer to obtain the final global features and

[0027] Furthermore, the detailed process of extracting local features in step 4 is as follows:

[0028] Step 4-3: Use a 1×1 convolution operation to reduce the dimension of the previously extracted shared features, reducing the number of channels to 1 / 4 of the original;

[0029] Step 4-4: Divide the shared features after dimensionality reduction into four equal parts according to the height of the image to obtain local feature groups and

[0030] Step 4-5: The four local feature blocks in the local feature group are adaptively averaged and input into the BN layer, and then spliced ​​according to the number of channels to obtain the final local feature. and Achieved unification with the global feature size.

[0031] Furthermore, in step 5, local features are used to supervise global features, so that the global features finally obtained can absorb the advantages of local features. Specifically, the present invention attaches local information to global features, and subsequently only uses global features to calculate the loss, which can be expressed as shown in Formula 1:

[0032]

[0033] By minimizing the local supervision loss, the spatial features of the image are transferred to the channel dimension of the global features, so that each group of channels contains the local information of a part of the area;

[0034] The global feature constraint in step 5 is composed of the identity loss and the cross-modal grouping loss newly designed by the present invention, wherein the identity loss can be expressed as Formula 2:

[0035]

[0036] Among them, p(y i |f i ) represents the global feature f i The model predicts that i probability;

[0037] The present invention designs the extracted global features of the two modal images, groups them according to the identity of the pedestrians, calculates the distance between the samples in the group, and the loss in the group can be expressed as Formula 3:

[0038]

[0039] Among them, f s and f tRepresents the global features of two samples in the group. By minimizing the intra-group loss, the intra-class distance across modalities is reduced. This not only reduces the difference within the modality, but also further alleviates the difference across modalities. The intra-group loss of the group is recorded as var.

[0040] For the groups, we hope to maximize the distance between the groups, that is, the distance between all pairs of samples in the two groups should be maximized. However, simply increasing the distance between the two groups may cause the samples to be scattered, and the distance between the samples in the groups may also increase. Therefore, further restrictions are added. While increasing the distance between the groups, the variance within each group remains unchanged, that is, the overall movement of the samples in the group is maintained. The loss between groups can be expressed as Formula 4:

[0041]

[0042] Among them, S i and S j represents two groups, i.e., two pedestrian identities, i and j are the labels of the two groups respectively, then and Represents a global feature in groups i and j respectively, var i and var j represents the intra-group distance of the two groups i and j; therefore, the total cross-modal grouping loss consists of two parts: the intra-group part and the inter-group part, which can be expressed as Formula 5:

[0043] L group =L intra +L inter (5);

[0044] Finally, the present invention proposes that the total loss can be expressed as shown in Formula 6:

[0045] L overall =L local +L id +L group (6).

[0046] Beneficial effects: The present invention proposes a cross-modal pedestrian re-identification method (LSN) using local supervision. First, the visible light image is converted into a grayscale image that is closer to the infrared image by using image processing methods, which alleviates the cross-modal differences at the image level and avoids the problem of introducing new noise and requiring additional training processes that may be caused by the use of GAN. Secondly, a two-stream network with shared parameters is used to extract discriminative shared features to alleviate cross-modal differences at the feature level; then, a local supervision network is designed so that the learned global features absorb the advantages of local features, enhance their robustness to background, occlusion and other noise, and alleviate internal differences in modalities; finally, a cross-modal grouping loss is designed, and the identity loss is used to constrain the network, while maintaining the intra-class distance, widening the inter-class distance and improving the overall performance. LSN takes into account both cross-modal differences and intra-modal differences. Experimental results show that on the SYSU-MM01 dataset, the recognition rate-related evaluation indicators Rank-1 and mAP reached 53.31% and 50.88% respectively; on the RegDB dataset, they reached 73.51% and 68.55%. The experimental results are better than similar methods, indicating the effectiveness and advancement of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a schematic diagram of the same pedestrian image under different cameras in the SYSU-MM01 dataset;

[0048] Figure 2 It is a schematic diagram of the visible light image and infrared image of a pedestrian;

[0049] Figure 3 It is a schematic diagram of the LSN network framework structure of the present invention;

[0050] Figure 4 It is a schematic diagram of the grayscale processing effect of the present invention;

[0051] Figure 5 It is a schematic diagram of the structure of the Nonlocal attention module of the present invention;

[0052] Figure 6 is a schematic diagram of the cross-modal grouping loss structure of the present invention;

[0053] Figure 7 It is a schematic diagram comparing the visual recognition performance of the method of the present invention and AGW on the SYSU-MM01 dataset;

[0054] Figure 8 This is a schematic diagram comparing the visual recognition performance of the proposed method and AGW in T2V mode on the RegDB dataset;

[0055] Fig. 9This is a schematic diagram comparing the visual recognition performance of this method and AGW in V2T mode on the RegDB dataset. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present invention will be described clearly and completely below so that those skilled in the art can better understand the advantages and features of the present invention, thereby making a clearer definition of the protection scope of the present invention. The embodiments described in the present invention are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0057] Example

[0058] This paper proposes a cross-modal person re-identification framework LSN (Local Supervision Network) using local supervision, the schematic diagram of which is shown in Figure 3 shown.

[0059] Assume that each time we train, we randomly select P pedestrian identities from the training set, and randomly select K visible light images and K infrared images for each identity. The following takes a visible light image and an infrared image as an example to introduce the training process. At this time, P and K are both 1, as follows:

[0060] Step 1: Input 1 visible light pedestrian image and 1 infrared pedestrian image to the LSN model and proceed to step 2;

[0061] Step 2: If it is an infrared pedestrian image, go to step 3; if it is a visible light pedestrian image, convert it into a grayscale image using image processing methods and go to step 3;

[0062] Step 3: Combine Resnet50 and Nonlocal attention modules to extract global shared features of infrared images and grayscale images, and proceed to step 4;

[0063] Step 4: Use the shared features to extract global features and local features respectively, and then proceed to step 5;

[0064] Step 5: Use local features to supervise global features, use cross-modal grouping loss combined with identity loss to constrain global features, and update network parameters through back propagation, and go to step 6;

[0065] Step 6: If the specified number of training rounds is reached, go to step 7, otherwise return to step 1;

[0066] Step 7: Training is complete.

[0067] In this embodiment, the pedestrian images in step 1 are all from the standard datasets SYSU-MM01 and RegDB for cross-modal pedestrian re-identification. Each pedestrian image is randomly cropped and randomly horizontally flipped to enhance the generalization ability of the experiment, and is unified into a size of 256×128. Assume represents the i-th visible light image, represents the jth infrared image, and Respectively and The corresponding pedestrian identity. Visible light modal sample and infrared modality samples Take for example to introduce.

[0068] In step 2, for pedestrian images in the visible light modality, the present invention uses grayscale images that are closer to infrared images as a transition. Before the visible light image is input into the network training, it is first converted into a grayscale image. This not only retains the structural information of the visible light image, but also reduces the dependence on color information during subsequent network training, alleviating cross-modal differences at the image level. The effect of grayscale processing is as follows: Figure 4 Compared with using GAN, this method does not require additional training, does not expand the network model, and does not increase training time, thereby improving the application efficiency in real scenarios; on the other hand, it avoids the introduction of additional noise and enhances the stability and reliability of the trained model. After conversion, the label information of the original sample is still retained, denoted as

[0069] The detailed process of step 3 is as follows:

[0070] Step 3-1: Pass the processed grayscale and infrared pedestrian images through a layer of convolutional network to extract the shallow-layer unique features of the two modes. and

[0071] Step 3-2: and The input is a network composed of the last four layers of Resnet50 and the Nonlocal attention module, and the shared features of the two modalities are extracted. and

[0072] In step 3-1, the convolutional networks of the two modalities have the same structure but different parameters. In step 3-2, the shallow-layer unique features of the two modalities are input into a network block with the same structure and parameters, and the features of the two different modalities are mapped to the same feature subspace to achieve feature alignment, thereby alleviating cross-modal differences at the feature level. This network block consists of the last four layers of the residual network Resnet50 and the Nonlocal attention module. The schematic diagram of the Nonlocal attention module is shown in the figure below. Figure 5 As shown in the figure, this module is used to refer to all locations and transmit long-distance information, making the extracted shared features more global and reliable.

[0073] In step 4, the detailed process of extracting global features is as follows:

[0074] Step 4-1: Perform adaptive average pooling on the shared features of the two modalities to obtain the preliminary global features and

[0075] Step 4-2: Input the prepared global features into the BN layer to obtain the final global features and On the one hand, it prevents the problems of gradient explosion and gradient disappearance, and on the other hand, it speeds up network training and convergence to prevent overfitting.

[0076] The detailed process of extracting local features is as follows:

[0077] Step 4-3: Use a 1×1 convolution operation to reduce the dimension of the previously extracted shared features and reduce the number of channels to 1 / 4 of the original.

[0078] Step 4-4: Divide the shared features after dimensionality reduction into four equal parts according to the height of the image to obtain local feature groups and

[0079] Step 4-5: The four local feature blocks in the local feature group are adaptively averaged and input into the BN layer, and then spliced ​​according to the number of channels to obtain the final local feature. and Achieved unification with the global feature size.

[0080] In step 5, local features are used to supervise global features, so that the final global features can absorb the advantages of local features and have stronger robustness to background clutter, occlusion and other noises, thereby alleviating internal differences in modalities and improving recognition performance. Different from the traditional method of using local features, the present invention does not calculate identity loss and triplet loss for each local feature, but attaches local information to the global feature, and only uses the global feature to calculate the loss later, which not only takes advantage of the local features but also reduces the amount of calculation. The loss can be expressed as shown in formula 1:

[0081]

[0082] By minimizing the local supervision loss, the spatial features of the image are transferred to the channel dimension of the global features, so that each group of channels contains the local information of a part of the area.

[0083] The global feature constraint in step 5 is composed of the identity loss and the cross-modal grouping loss newly designed by the present invention. The identity loss can be expressed as formula 2:

[0084]

[0085] Among them, p(y i |f i ) represents the global feature f i The model predicts that i probability.

[0086] The schematic diagram of cross-modal grouping loss is as follows Figure 6 , where the small balls represent the global features extracted by the above network, each color represents a pedestrian identity, and the solid and shadow represent the grayscale and infrared modes respectively.

[0087] The present invention designs the extracted global features of the two modal images, groups them according to the identity of the pedestrians, calculates the distance between the samples in the group, and the loss in the group can be expressed as Formula 3:

[0088]

[0089] Among them, f s and f t It represents the global features of two samples in the group. By minimizing the intra-group loss, the intra-class distance across modalities is reduced. This not only reduces the internal differences within the modality, but also further alleviates the cross-modal differences. The intra-group loss of the group is recorded as var.

[0090] For the groups, we hope to maximize the distance between the groups, that is, the distance between all pairs of samples in the two groups should be maximized. However, simply increasing the distance between the two groups may cause the samples to be scattered, and the distance between the samples within the groups may also increase. Therefore, further restrictions are added. While increasing the distance between the groups, the variance within each group remains unchanged, that is, the overall movement of the samples within the group is maintained. The loss between groups can be expressed as Formula 4:

[0091]

[0092] Among them, S i and S j represents two groups, i.e., two pedestrian identities, i and j are the labels of the two groups respectively, then and Represents a global feature in groups i and j respectively, var i and var j represents the intra-group distance of the two groups i and j. Therefore, the total cross-modal grouping loss consists of two parts: intra-group and inter-group, which can be expressed as Formula 5:

[0093] L group =L intra +L inter (5);

[0094] Finally, the present invention proposes that the total loss can be expressed as shown in Formula 6:

[0095] L overall =L local +L id +L group (6).

[0096] The present invention adopts the PyTorch framework and performs training and testing on a single NVIDIA GeForce 1080Ti GPU. The present invention adopts ResNet50 as the backbone network and initializes the network weights with parameters pre-trained on ImageNet. During the training process, 8 pedestrians are randomly selected each time, and then 4 visible light images and 4 infrared images are randomly selected for each pedestrian identity, and the batchsize is 64. This experiment sets the initial learning rate to 0.01, optimizes by stochastic gradient descent, sets the momentum parameter to 0.9, and uses the warm-up learning rate strategy in the first 10 rounds. A total of 80 rounds of training are conducted, and the learning rate changes with the number of training rounds as shown in Formula 7:

[0097]

[0098] For the sake of fairness, referring to the existing methods, this paper also uses Cumulative Matching Characteristic (CMC) and mean Average Precision (mAP) as evaluation indicators. Rank-k in CMC measures the probability of correct cross-modal pedestrian images appearing in the first k retrieval results, while mAP can reflect the average retrieval performance of the method. Specific embodiment 1:

[0100] This embodiment will use the SYSU-MM01 public dataset to complete the cross-modal pedestrian re-identification task and test the performance of the model. The SYSU-MM01 dataset is the first standard dataset in the field of cross-modal pedestrian re-identification, which is collected by 4 visible light cameras and 2 infrared cameras. The training data of this dataset consists of 22,258 visible light images and 11,909 infrared images of 395 pedestrians, and each pedestrian is captured by at least two cameras with different perspectives and positions. The test set is divided into two evaluation modes and includes an additional 95 pedestrians. In both evaluation modes, the query set is consistent, containing 3,803 infrared images captured by two infrared cameras. In the All-Search mode, the gallery set contains images captured by 4 visible light cameras, and in the Indoor-Search mode, it only contains images captured by 2 indoor visible light cameras. In this experiment, the most stringent evaluation method is used in the SYSU-MM01 dataset. The test set is constructed using the Single-Shot method, and the methods are evaluated in two evaluation modes: All-Search and Indoor-Search. The comparative experimental results of the proposed method and other existing methods on the SYSU-MM01 dataset are shown in Tables 1 and 2.

[0101] In the comparative experiment, in addition to the solution of the present invention, the best effect is the AGW algorithm, which uses the non-local attention mechanism to extract global features in the feature extraction stage, and uses generalized average pooling and weighted regularized triplet loss for constraints. On this basis, the present invention gives full play to the advantages of local features, uses local features to supervise global features, so that the learned global features inherit the robustness of local features to noise such as background clutter and occlusion, alleviates the internal differences of the modalities, and designs a new cross-modal grouping loss, which is combined with the identity loss to constrain the network and improve the overall performance of the network. The experimental results show that compared with AGW, the method of the present invention improves Rank-1, Rank-10, and mAP by 6.01%, 6.09%, and 3.23% in All-Search mode; in Indoor-Search mode, Rank-1, Rank-10, and mAP are increased by 5.04%, 2.75%, and 3.21%, respectively.

[0102] Table 1 Comparison results of the proposed method and other methods in All-Search mode on the SYSU-MM01 dataset

[0103]

[0104]

[0105] Table 2 Comparison results of the proposed method and other methods in the Indoor-Search mode on the SYSU-MM01 dataset

[0106]

[0107]

[0108] To further demonstrate the advanced nature of the method of the present invention, we selected the suboptimal solution AGW for comparison and visualized the experimental results. Three images of different pedestrians taken from different perspectives were selected as representatives, where query image 1 is a frontal image of a boy with a relatively clear image of his shirt, query image 2 is a side image of a boy with obvious occlusion of the body by a shoulder bag and interference from background noise such as a table, and query image 3 is a back image of a girl with a clear body outline but no identifiable features visible to the human eye. The retrieval results are as follows: Figure 7 As shown, a green box indicates a correct match, and a red box indicates an incorrect match.

[0109] The visualization results also further verify the effectiveness and advancement of the method proposed in this invention. Specific embodiment 2:

[0111] This embodiment uses the RegDB dataset to complete the cross-modal pedestrian re-identification task and test the performance of the model. The RegDB dataset is a small-scale dataset consisting of images captured by a visible light camera and a far infrared camera. The dataset contains 412 pedestrians, each of which contains 10 visible light images and 10 infrared images. This experiment randomly selects 2060 images corresponding to 206 pedestrians for training, and the remaining 2060 images corresponding to the remaining 206 pedestrians are used for testing. This experiment uses two retrieval modes: visible light image retrieval infrared image (Visible-to-Thermal, V2T) and infrared image retrieval visible light image (Thermal-to-Visible, T2V), and uses 10 random splits to select training sets and test sets, and record the average accuracy as the final performance on the dataset. The comparative experimental results are shown in Tables 3 and 4.

[0112] Table 3 Comparison results of the proposed method and other methods in T2V mode on the RegDB dataset

[0113]

[0114] Table 4 Comparison results of the proposed method and other methods in V2T mode on the RegDB dataset

[0115]

[0116]

[0117] It can be seen that on the RegDB dataset, the proposed method is also ahead of the existing methods in various indicators. Compared with the second-best method AGW, in the T2V mode, Rank-1, Rank-10, and mAP are improved by 3.02%, 1.84%, and 2.65%, respectively; in the V2T mode, Rank-1, Rank-10, and mAP are improved by 2.80%, 1.42%, and 2.56%, respectively.

[0118] Similarly, we also select AGW as a comparison method and visualize the experimental results on the RegDB dataset. For the two retrieval modes T2V and V2T, we randomly select two different pedestrian images as the query set, and the results are as follows: Figure 8 , Fig. 9 As shown. Among them, Figure 8 In the example, the query image is an infrared image, lacking information such as color and texture, and recognition mainly relies on the posture of pedestrians. Fig. 9In the example, the query image is a visible light image, which contains rich color and texture information, but also contains complex background and occlusion noise. The visualization experiment results also show that the method of the present invention performs better than the AGW algorithm in both matching modes.

[0119] The references involved in Tables 1 to 4 of the present invention are as follows:

[0120] [1]Wang G, Zhang T, Cheng J, et al. Rgb-infrared cross-modality personre-identification via joint pixel and feature alignment[C] / / Proc of IEEE / CVFInternational Conference on Computer Vision. Piscataway, NJ: IEEE Press, 2019: 3622-3631.

[0121] [2]Ye M, Shen J, Lin G, et al. Deep learning for person re-identification: a survey and outlook[J]. IEEE Trans on Pattern Analysis and Machine Intelligence, 2021, 44(6): 2872-2893.

[0122] [3]Wu A, Zheng W, Yu H, et al. Rgb-infrared cross-modality person re-identification[C] / / Proc of IEEE Conference International Conference on Computer Vision. Piscataway, NJ: IEEE Press, 2017: 5390-5399.

[0123] [4]Ye M,Wang Z,Lan X,et al.Visible thermal person re-identificationvia dual-constrained top-ranking[C] / / Proc of the 27th International JointConference on Artificial Intelligence.Palo Alto,CA:AAAI Press,2018:1092–1099。

[0124] [5]Wang Z,Wang Z,Zheng Y,et al.Learning to reduce dual-leveldiscrepancy for infrared-visible person re-identification[C] / / Proc of IEEE / CVF Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEEPress,2019:618-626。

[0125] [6]Ye M,Lan X,Li J,et al.Hierarchical discriminative learning forvisible thermal person re-identification[C] / / Proc of the 2018 AAAI Conferenceon Artificial Intelligence.Palo Alto,CA:AAAI Press,2018:7501-7508。

[0126] [7]Dai P,Ji R,Wang H,et al.Cross-modality person re-identificationwith generative adversarial training[C] / / Proc of the 27th International JointConference on Artificial Intelligence.Palo Alto,CA:AAAI Press,2018:677-683.。

[0127] [8]Ye M,Lan X,Wang Z,et al.Bi-directional center-constrained top-ranking for visible thermal person re-identification[J].IEEE Trans onInformation Forensics and Security,2019,15:407-419。

[0128] [9]Hao Y,Wang N,Li J,et al.HSME:hypersphere manifold embedding forvisible thermal person re-identification[C] / / Proc of the 2019 AAAI Conferenceon Artificial Intelligence.Palo Alto,CA:AAAI Press,2019:8385-8392。

[0129]

[10] Choi S,Lee S,Kim Y,et al.Hi-CMD:hierarchical cross-modalitydisentanglement for visible-infrared person re-identification[C] / / Proc ofIEEE / CVF Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE Press,2020:10254–10263。

[0130]

[11] Wu A,Zheng W,Gong S,et al.Rgb-ir person re-identification bycross-modality similarity preservation[J].International Journal of ComputerVision,2020,128(6):1765-1785。

[0131]

[12] Ye M, Shen J, Crandall DJ, et al. Dynamic dual-attentive aggregation learning for visible-infrared person re-identification[C] / / Proc of European Conference on Computer Vision. Berlin, German: Springer Press, 2020: 229–247. Specific embodiment 3:

[0133] This embodiment will introduce an applicable scenario of the present invention.

[0134] One day, a theft case occurred in a shopping mall. The police locked the crime scene based on the information provided by the reporter and locked the suspect from the surveillance video recorded by the corresponding camera. At this time, the suspect's normal visible light image in the shopping mall was obtained. However, after the suspect succeeded in committing the crime, he would often hide or flee in the shadow of the crowd and choose a place with poor light conditions. At this time, the surveillance at the corresponding location obtained the infrared pedestrian image taken by the infrared camera, which brought great difficulty to find the suspect. The solution proposed by the present invention can solve this problem.

[0135] First, the police obtain the surveillance video of the area that needs to be queried, use pedestrian detection technology to crop out the images of pedestrians who appear, and name them according to the time and camera number.

[0136] Secondly, these pedestrian images are used as a gallery set and the suspect images obtained in advance are used as a query set, which are input into the network model designed by the present invention.

[0137] Finally, the system will spit out a set of image sequences. Based on the name of this set of image sequences (time, camera number), the suspect's activity trajectory can be locked for the police to arrest.

Claims

1. A cross-modal person re-identification method using local supervision, Features: A cross-modal person re-identification framework LSN using local supervision is established. Assume that each time training, P pedestrian identities are randomly selected in the training set, and K visible light images and K infrared images are randomly selected for each identity, as follows: Step 1: Input K visible light pedestrian images and K infrared pedestrian images to the LSN model and proceed to step 2; Step 2: If it is an infrared pedestrian image, go to step 3; if it is a visible light pedestrian image, convert it into a grayscale image using image processing methods and go to step 3; Step 3: Combine Resnet50 and Nonlocal attention modules to extract global shared features of infrared images and grayscale images, and proceed to step 4; Step 4: Use the shared features to extract global features and local features respectively, and then proceed to step 5; Step 5: Use local features to supervise global features, use cross-modal grouping loss combined with identity loss to constrain global features, and update network parameters through back propagation, and go to step 6; Step 6: If the specified number of training rounds is reached, go to step 7, otherwise return to step 1; Step 7: Training ends; in: In step 5, local features are used to supervise global features, so that the final global features can absorb the advantages of local features. Specifically, local information is attached to global features, and only global features are used to calculate the loss later. The loss can be expressed as shown in formula 1: By minimizing the local supervision loss, the spatial features of the image are transferred to the channel dimension of the global features, so that each group of channels contains the local information of a part of the area; The global feature constraint in step 5 is composed of identity loss and cross-modal grouping loss, where the identity loss can be expressed as formula 2: Among them, p(y i |f i ) represents the global feature f i The model predicts that i probability; The global features of the two modal images extracted are grouped according to the identity of the pedestrians, and the distance between samples in the group is calculated. The loss within the group can be expressed as Formula 3: Among them, f s and f t Represents the global features of two samples in the group. By minimizing the intra-group loss, the intra-class distance across modalities is reduced, which not only reduces the difference within the modality, but also further alleviates the difference across modalities. For the groups, we hope to maximize the distance between the groups, that is, the distance between all pairs of samples in the two groups should be maximized. However, simply increasing the distance between the two groups may cause the samples to be scattered, and the distance between the samples in the groups may also increase. Therefore, further restrictions are added. While increasing the distance between the groups, the variance within each group remains unchanged, that is, the overall movement of the samples in the group is maintained. The loss between groups can be expressed as Formula 4: Among them, S i and S j represents two groups, i.e., two pedestrian identities, i and j are the labels of the two groups respectively, then and Represents a global feature in groups i and j respectively, var i and var j represents the intra-group distance of the two groups i and j; therefore, the total cross-modal grouping loss consists of two parts: the intra-group part and the inter-group part, which can be expressed as Formula 5: L group =L intra +L inter (5); Finally, the total loss can be expressed as shown in Formula 6: L overall =L local +L id +L group (6); The pedestrian images in step 1 are all from the standard datasets SYSU-MM01 and RegDB for cross-modal pedestrian re-identification; each pedestrian image is randomly cropped and randomly horizontally flipped to enhance the generalization ability of the experiment, and unified into a size of 256×128; assuming represents the i-th visible light image, represents the jth infrared image, and Respectively and The corresponding pedestrian identity; among them, input the same pedestrian The visible light modal sample is The infrared modality samples are In step 2, for the pedestrian image in the visible light mode, a grayscale image that is closer to the infrared image is selected as a transition. Before the visible light image is input into the network training, it is first converted into a grayscale image. After the conversion, the label information of the original sample is still retained, which is recorded as The detailed process of step 3 is as follows: Step 3-1: Pass the processed grayscale and infrared pedestrian images through a layer of convolutional network to extract the shallow-layer unique features of the two modes. and Step 3-2: and The input is a network composed of the last four layers of Resnet50 and the Nonlocal attention module, and the shared features of the two modalities are extracted. and The detailed process of extracting global features in step 4 is as follows: Step 4-1: Perform adaptive average pooling on the shared features of the two modalities to obtain the preliminary global features and Step 4-2: Input the prepared global features into the BN layer to obtain the final global features and The detailed process of extracting local features in step 4 is as follows: Step 4-3: Use a 1×1 convolution operation to reduce the dimension of the previously extracted shared features, reducing the number of channels to 1 / 4 of the original; Step 4-4: Divide the shared features after dimensionality reduction into four equal parts according to the height of the image to obtain local feature groups and Step 4-5: The four local feature blocks in the local feature group are adaptively averaged and input into the BN layer, and then spliced ​​according to the number of channels to obtain the final local feature. and Achieved unification with the global feature size.

2. The cross-modal person re-identification method using local supervision according to claim 1, Features: In the step 3-1, the convolutional network structures of the two modes are the same but the parameters are different.

3. The cross-modal person re-identification method using local supervision according to claim 1, Features: In the step 3-2, the shallow-layer specific features of the two modalities are input into a network block with the same structure and parameters, and the features of the two different modalities are mapped to the same feature subspace to achieve feature alignment, thereby alleviating cross-modal differences at the feature level; the network block consists of the last four layers of the residual network Resnet50 and the Nonlocal attention module.

Citation Information

Patent Citations

  • Cross-modal pedestrian re-identification method based on global level and local level joint constraint

    CN114550208A