A cross-modal person re-identification method and system based on multi-feature learning
By using a dual-stream network to extract global and local features of visible light and infrared images in cross-modal pedestrian re-identification and constructing an overall loss function, the problem of low recognition accuracy in the prior art is solved, and the recognition accuracy and robustness are improved .
Patent Information
- Application Number
- CN202210135424.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-14
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2042-02-14
AI Technical Summary
The existing cross-modal pedestrian re-identification method only uses global or local features of the image, resulting in low recognition accuracy and inability to make full use of information, affecting the model's detection accuracy of target pedestrians in the image library.
The dual-stream network is used to extract the global and local features of visible light and infrared images, calculate the difficult five-tuple loss and global identity loss, and build the overall loss function based on specific identity loss, and optimize the cross-modal pedestrian re-identification model to enhance feature extraction information capacity and discernibility.
By combining global and local features, the feature discrimination and robustness of the model are enhanced, the recognition accuracy of cross-modal pedestrian re-identification is improved, and the problem of insufficient single feature extraction is compensated.
Smart Images
Figure CN114495010B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image recognition, and more specifically, to a cross-modal pedestrian re-identification method and system based on multi-feature learning. Background Art
[0002] Person re-identification refers to finding people with the same identity in different images captured by different cameras by matching people given a picture of a pedestrian to be identified. The visible light-infrared image cross-modal pedestrian re-identification method can match pedestrian images of the two modalities with each other, making full use of the monitoring data captured by visible light and infrared cameras to achieve the purpose of 24-hour monitoring.
[0003] There is a cross-modal pedestrian re-identification method and system based on a two-stream convolutional neural network. It uses the feature extraction parts of two convolutional neural networks and adopts the cross entropy loss function, cross-modal triplet loss function, relative entropy loss function and random permutation strategy to train the two convolutional neural networks on a cross-modal dataset, extracts the features of the two modal images respectively, and then performs Euclidean distance measurement on the features of the two modalities to obtain feature matching results, thereby realizing cross-modal pedestrian re-identification.
[0004] However, visible light images and infrared images are not only different between modalities, but even within the same modality, due to changes in pedestrian posture, monitoring angle of view and light intensity, there will be many differences between images of the same identity in the same modality. The above method only uses the global features of the two modal images for pedestrian re-identification. The method of using only single feature information makes the model insufficient in extracting information from the image, which may cause the model to be overly dependent on some information, affecting the accuracy of the model in detecting the target pedestrian from the image library. Summary of the invention
[0005] In order to overcome the defect in the prior art of using only global features or local features of an image for cross-modal pedestrian re-identification, resulting in low recognition accuracy, the present invention provides a cross-modal pedestrian re-identification method and system based on multi-feature learning.
[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:
[0007] In a first aspect, the present invention proposes a cross-modal person re-identification method based on multi-feature learning, comprising the following steps:
[0008] S1: Build a cross-modal person re-identification model including a two-stream network;
[0009] S2: Randomly obtain several batches of visible light images and infrared images of pedestrians to construct training sets and test sets;
[0010] S3: inputting the visible light image and the infrared image into a cross-modal person re-identification model in batches, respectively, and the cross-modal person re-identification model outputs global features of the visible light image and the infrared image, and local features of the visible light image and the infrared image;
[0011] S4: Calculate the difficult quintuple loss and the global identity loss according to the global features, and calculate the specific identity loss according to the local features;
[0012] S5: constructing an overall loss function of the cross-modal person re-identification model according to the difficult quintuple loss, the global identity loss and the specific identity loss, and optimizing parameters of the cross-modal person re-identification model according to the overall loss function;
[0013] S6: Input the images in the test set and the target images to be identified into the optimized cross-modal person re-identification model, calculate the feature similarity between the target images to be identified and the images in the test set, and output the first M images in the test set with the highest feature similarity to the target images to be identified as the person re-identification results.
[0014] Preferably, in S3, several batches of visible light images and infrared images of pedestrians are randomly acquired, each batch including P visible light images and P infrared images of N types of pedestrians, for a total of 2NP images.
[0015] The three-dimensional tensors of the visible light image and the infrared image are globally averaged and pooled to obtain the global features of the visible light image and the infrared image.
[0016] The three-dimensional tensors of the visible light image and the infrared image are horizontally divided into K horizontal partial tensors, and the K horizontal partial tensors are globally average pooled to obtain K column vectors; a convolutional layer is used to reduce the dimension of the K column vectors to obtain local features of the visible light image and the infrared image.
[0017] Preferably, in S4, the difficult five-tuple loss L is calculated based on the global features of the visible light image and the infrared image. HP , and its calculation formula is as follows:
[0018]
[0019] Where N represents the number of pedestrian categories, P represents the number of visible light images or infrared images, and L hgt represents the difficult global triplet loss, L hct Representation difficulty cross-modal triplet loss;
[0020] The difficult global triplet loss L hgt The calculation formula is as follows:
[0021]
[0022] Where p = 1…2P, a = 1…2P, p≠a, n = 1…2P, i = 1…N, j = 1…N, j≠i, α represents the difference between the maximum Euclidean distance and the minimum Euclidean distance between images; f i a represents the global features of the a-th image of pedestrian category i, f i p represents the global features of the pth image of pedestrian category i, represents the global features of the nth image of pedestrian category j; represents the maximum Euclidean distance between the ath image of pedestrian category i and the nth image of pedestrian category i, represents the minimum Euclidean distance between the a-th image of pedestrian category i and the n-th image of pedestrian category j; function [*] + If * is greater than 0, [*] + Output *, if * is less than or equal to 0, [*] + The output is 0;
[0023] The difficult cross-modal triplet loss L hct The calculation formula is as follows:
[0024]
[0025] Among them, f i cp Indicates that the pedestrian category is i and is the same as f i a Global features that do not belong to the same modality image, Indicates that the pedestrian category is j and is the same as f i a Global features that do not belong to the same modality image; represents the minimum Euclidean distance between the image of pedestrian category i and the image of pedestrian category j in another modality, maxd(f i a ,f i cp ) represents the maximum Euclidean distance between the a-th image of pedestrian category i and the image of pedestrian category i in the other modality.
[0026] Preferably, in S4, a global classifier is used to classify the global features, and a cross entropy loss is calculated as the global identity loss L id , and its calculation formula is as follows:
[0027]
[0028] in, represents the label of the i-th visible light image, V represents the visible light image, Represents the probability result of the global classifier classifying the i-th visible light image; is the label of the i-th infrared image, I represents the infrared image, Represents the probability result of the global classifier classifying the i-th infrared image.
[0029] Preferably, in S4, a visible light classifier is used to classify the local features of the visible light image, and an infrared classifier is used to classify the local features corresponding to the infrared image, and a cross entropy loss is calculated as the specific identity loss L sid , and its calculation formula is as follows:
[0030]
[0031] in, represents the probability result of the visible light classifier classifying the jth local feature of the i-th visible light image, It represents the probability result of the infrared classifier classifying the jth local feature of the i-th infrared image.
[0032] According to the difficult quintuple loss L HP , global identity loss L id and the identity-specific loss L sid The overall loss function L of the cross-modal person re-identification model is constructed, and its expression is as follows:
[0033] L=L id +L HP +λL sid
[0034] Among them, λ is the weight parameter of the loss function.
[0035] Preferably, in S1, the dual-stream network includes a visible light modality network and an infrared modality network;
[0036] The visible light modality network includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a first horizontal segmentation module, a first pooling layer, a visible light classifier and a first global classifier; the first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module are connected in sequence, and the output end of the fifth convolution module is connected to the input end of the first horizontal segmentation module and the first pooling layer respectively; the output end of the first horizontal segmentation module is connected to the input end of the sixth convolution module, and the output end of the sixth convolution module is connected to the input end of the visible light classifier; the output end of the first pooling layer is connected to the input end of the first global classifier;
[0037] The infrared modality network includes a seventh convolution module, an eighth convolution module, a ninth convolution module, a tenth convolution module, an eleventh convolution module, a twelfth convolution module, a second horizontal segmentation module, a second pooling layer, an infrared classifier and a second global classifier; the seventh convolution module, the eighth convolution module, the ninth convolution module, the tenth convolution module and the eleventh convolution module are connected in sequence, and the output end of the eleventh convolution module is connected to the input end of the second horizontal segmentation module and the second pooling layer respectively; the output end of the second horizontal segmentation module is connected to the input end of the twelfth convolution module, and the output end of the twelfth convolution module is connected to the input end of the infrared classifier; the output end of the second pooling layer is connected to the input end of the second global classifier;
[0038] The third convolution module shares parameters with the ninth convolution module; the fourth convolution module shares parameters with the tenth convolution module; the fifth convolution module shares parameters with the eleventh convolution module; the second convolution module shares parameters with the eighth convolution module;
[0039] Parameters of the first global classifier and the second global classifier are shared.
[0040] Preferably, the visible light modality network and the infrared modality network use BotNet as the backbone network.
[0041] Preferably, the visible light classifier, the first global classifier, the infrared classifier and the second global classifier each include a fully connected layer and a normalization layer.
[0042] In a second aspect, the present invention further proposes a cross-modal pedestrian re-identification system based on multi-feature learning, which is applied to the cross-modal pedestrian re-identification method based on multi-feature learning described in any of the above schemes, comprising:
[0043] The data acquisition module is used to randomly acquire several batches of visible light images and infrared images of pedestrians to construct training sets and test sets;
[0044] The cross-modal person re-identification model includes a two-stream network, which is used to output the global features of the visible light image and the infrared image, as well as the local features of the visible light image and the infrared image according to the acquired visible light image and the infrared image; and calculate the similarity between the target image to be identified and the image in the test set, and output the first M images in the test set with the highest similarity to the target image to be identified as the person re-identification result;
[0045] The optimization module is used to calculate the loss function of the cross-modal pedestrian re-identification model including the difficult quintuple loss, the global identity loss and the local feature calculation specific identity loss according to the global features of the visible light image and the infrared image, as well as the local features of the visible light image and the infrared image, and optimize the parameters of the cross-modal pedestrian re-identification model according to the loss function.
[0046] Compared with the prior art, the beneficial effects of the technical solution of the present invention are as follows: the present invention uses a two-stream network to extract global features and local features of each batch of visible light images and infrared images, and calculates the difficult five-tuple loss based on the global features, performs identity classification for the global features and each local feature, and obtains the global identity loss and the specific identity loss; constructs the overall loss function of the cross-modal pedestrian re-identification model based on the difficult five-tuple loss, the global identity loss and the specific identity loss, and optimizes the parameters of the cross-modal pedestrian re-identification model based on the overall loss function. Among them, the specific identity loss enables the cross-modal pedestrian re-identification model to notice the subtle differences between images of different pedestrian categories, which is conducive to extracting the discriminative features of a specific identity. The present invention takes into account the global features and local features of visible light images and infrared images, and enhances the information capacity of feature extraction by combining global features and local features, making up for the shortcomings caused by using only one feature for pedestrian re-identification, enhancing the discriminability of model features, and improving the recognition accuracy of cross-modal pedestrian re-identification. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 Flowchart of the cross-modal person re-identification method based on multi-feature learning.
[0048] Figure 2 This is a schematic diagram of the dual-flow network in Example 2.
[0049] Figure 3 The architecture diagram of the cross-modal pedestrian re-identification system based on multi-feature learning. DETAILED DESCRIPTION
[0050] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;
[0051] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0052] Example 1
[0053] See also Figure 1 , this embodiment proposes a cross-modal pedestrian re-identification method based on multi-feature learning, including the following steps:
[0054] S1: Build a cross-modal person re-identification model including a two-stream network;
[0055] S2: Randomly obtain several batches of visible light images and infrared images of pedestrians to construct training sets and test sets;
[0056] S3: inputting the visible light image and the infrared image into a cross-modal person re-identification model in batches, respectively, and the cross-modal person re-identification model outputs global features of the visible light image and the infrared image, and local features of the visible light image and the infrared image;
[0057] S4: Calculate the difficult quintuple loss and the global identity loss according to the global features, and calculate the specific identity loss according to the local features;
[0058] S5: constructing an overall loss function of the cross-modal person re-identification model according to the difficult quintuple loss, the global identity loss and the specific identity loss, and optimizing parameters of the cross-modal person re-identification model according to the overall loss function;
[0059] S6: Input the images in the test set and the target images to be identified into the optimized cross-modal person re-identification model, calculate the feature similarity between the target images to be identified and the images in the test set, and output the first M images in the test set with the highest feature similarity to the target images to be identified as the person re-identification results.
[0060] In the specific implementation process, visible light images and infrared images are input into the cross-modal pedestrian re-identification model in batches, and the two-stream network in the cross-modal pedestrian re-identification model outputs three-dimensional tensors of visible light images and infrared images. On the one hand, the three-dimensional tensors of visible light images and infrared images are globally averaged pooled to obtain global features of visible light images and infrared images; on the other hand, the three-dimensional tensors of visible light images and infrared images are horizontally split to obtain corresponding horizontal part tensors, and the horizontal part tensors are globally averaged pooled to obtain corresponding column vectors, and then the column vectors are reduced in dimension using a convolutional layer to obtain local features of visible light images and infrared images.
[0061] The difficult five-tuple loss is calculated based on the global features of the visible light image and the infrared image, and the identity is classified for the global features and each local feature of the visible light image and the infrared image to obtain the global identity loss and the specific identity loss, so as to construct the overall loss function of the cross-modal pedestrian re-identification model, and the parameters of the cross-modal pedestrian re-identification model are optimized according to the overall loss function. Finally, the images in the test set and the target images to be identified are input into the optimized cross-modal pedestrian re-identification model, and the feature similarity between the target images to be identified and the images in the test set is calculated. The first 20 images in the test set with the highest feature similarity to the target images to be identified are output as the pedestrian re-identification results.
[0062] Not only are visible light images and infrared images different from each other in terms of modality, but even within the same modality, changes in pedestrian posture, monitoring angle of view, and light intensity can also lead to many differences between images of the same identity in the same modality.
[0063] The present invention takes into account the global features and local features of visible light images and infrared images, and enhances the information capacity of feature extraction by combining global features and local features, making up for the deficiency of using only one feature for pedestrian re-identification, enhancing the discriminability and robustness of model features, and improving the recognition accuracy of cross-modal pedestrian re-identification.
[0064] Example 2
[0065] This embodiment proposes a cross-modal pedestrian re-identification method based on multi-feature learning, including the following steps:
[0066] S1: Build a cross-modal person re-identification model including a two-stream network.
[0067] S2: Randomly obtain several batches of visible light images and infrared images of pedestrians. Each batch includes P visible light images and P infrared images of N types of pedestrians, a total of 2NP images to construct training sets and test sets.
[0068] In this embodiment, several batches of visible light images and infrared images of pedestrians are randomly obtained from the SYSU-MM01 dataset to construct a training set and a test set, wherein the infrared images are selected as the image set to be retrieved in the test set, and the visible light images are used as the retrieval library; the images in the test set are reset to a size of 288*144, 10 pixels are padded on the edges of the images in the training set, and then they are randomly cropped to obtain images of a size of 288*144, and the images are horizontally flipped with a probability of 0.5.
[0069] S3: Inputting the visible light image and the infrared image into a cross-modal person re-identification model in batches, respectively, and the cross-modal person re-identification model outputs global features of the visible light image and the infrared image, as well as local features of the visible light image and the infrared image.
[0070] In this embodiment, 8 pedestrian categories are selected from each batch of visible light images and infrared images of pedestrians, and 4 visible light images and infrared images are selected for each pedestrian category, that is, N = 8, P = 4. The visible light images and infrared images of each batch in the training set are respectively input into the cross-modal pedestrian re-identification model.
[0071] In this embodiment, the dual-stream network includes a visible light modality network and an infrared modality network. Figure 2 As shown, Figure 2 Schematic diagram of the two-stream network proposed for this implementation.
[0072] The visible light modality network includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a first horizontal segmentation module, a first pooling layer, a visible light classifier and a first global classifier; the first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module are connected in sequence, and the output end of the fifth convolution module is connected to the input end of the first horizontal segmentation module and the first pooling layer respectively; the output end of the first horizontal segmentation module is connected to the input end of the sixth convolution module, and the output end of the sixth convolution module is connected to the input end of the visible light classifier; the output end of the first pooling layer is connected to the input end of the first global classifier.
[0073] The infrared modality network includes a seventh convolution module, an eighth convolution module, a ninth convolution module, a tenth convolution module, an eleventh convolution module, a twelfth convolution module, a second horizontal segmentation module, a second pooling layer, an infrared classifier and a second global classifier; the seventh convolution module, the eighth convolution module, the ninth convolution module, the tenth convolution module and the eleventh convolution module are connected in sequence, and the output end of the eleventh convolution module is respectively connected to the input end of the second horizontal segmentation module and the second pooling layer; the output end of the second horizontal segmentation module is connected to the input end of the twelfth convolution module, and the output end of the twelfth convolution module is connected to the input end of the infrared classifier; the output end of the second pooling layer is connected to the input end of the second global classifier.
[0074] The first convolution module and the fifth convolution module do not share parameters, and the first convolution layer and the third convolution layer do not share parameters, and are used to extract specific features of visible light and infrared modalities respectively.
[0075] The parameters of the third convolution module are shared with the ninth convolution module; the parameters of the fourth convolution module are shared with the tenth convolution module; the parameters of the fifth convolution module are shared with the eleventh convolution module; the parameters of the second convolution module are shared with the eighth convolution module, and the parameters of the first global classifier and the second global classifier are shared, which are used to extract common features between the visible light and infrared modalities.
[0076] The first convolution module and the seventh convolution module do not share parameters, and the sixth convolution layer and the twelfth convolution layer do not share parameters, and are used to extract specific features of visible light and infrared modalities respectively.
[0077] The visible light classifier and the infrared classifier do not share parameters and are used to classify specific local features of different modalities.
[0078] In this embodiment, the visible light modality network and the infrared modality network use BotNet pre-trained on ImageNet as the backbone network. In this embodiment, BoTNet based on the idea of attention is used as the dual-stream network of the backbone network to extract global features. The fifth convolution module and the eleventh convolution module replace the convolution layer with a convolution kernel of 3×3 with a multi-head self-attention layer (MHSA), which reduces network parameters and enhances the global information aggregation capability of the network architecture.
[0079] In this embodiment, the visible light classifier, the first global classifier infrared classifier and the second global classifier each include a fully connected layer and a normalization layer.
[0080] In this embodiment, the visible light image is input into the cross-modal pedestrian re-identification model, and the visible light image first passes through the first convolution module, the second convolution module, the third convolution module, the fourth convolution module, and the fifth convolution module, and the fifth convolution module outputs the three-dimensional tensor T1 of the visible light image. On the one hand, the three-dimensional tensor T1 of the visible light image is subjected to global average pooling through the first pooling layer to obtain the global features of the visible light image; on the other hand, the three-dimensional tensor T1 of the visible light image is subjected to the first horizontal segmentation module, and is horizontally segmented into 6 horizontal partial tensors, and the 6 horizontal partial tensors are subjected to global average pooling to obtain 6 column vectors; the sixth convolution module is used to reduce the dimensionality of the 6 column vectors to obtain the local features of the visible light image.
[0081] The infrared image is input into the cross-modal pedestrian re-identification model. The infrared image first passes through the seventh convolution layer, the eighth convolution module, the ninth convolution module, the tenth convolution module and the eleventh convolution module. The eleventh convolution module outputs the three-dimensional tensor T2 of the infrared image. On the one hand, the three-dimensional tensor T2 of the infrared image is subjected to global average pooling through the second pooling layer to obtain the global features of the infrared image; on the other hand, the three-dimensional tensor T2 of the infrared image is horizontally divided into 6 horizontal part tensors through the first horizontal segmentation module, and the 6 horizontal part tensors are subjected to global average pooling to obtain 6 column vectors; the twelfth convolution module is used to reduce the dimension of the 6 column vectors to obtain the local features of the infrared image.
[0082] S4: Calculate the difficult quintuple loss and the global identity loss according to the global features, and calculate the specific identity loss according to the local features.
[0083] In this embodiment, the global features of visible light images and infrared images in each batch are combined to calculate the difficult quintuple loss L HP , the difficult quintuple loss L HP By the difficult global triplet loss L hgt and the difficult cross-modal triplet loss L hct composition.
[0084] Difficult global triplet loss L hgt The calculation formula is as follows:
[0085]
[0086] Wherein, p = 1…2P, a = 1…2P, p≠a, n = 1…2P, i = 1…N, j = 1…N, j≠i, α represents the difference between the maximum Euclidean distance and the minimum Euclidean distance between images, and the value of α is 0.9; f i a represents the global features of the a-th image of pedestrian category i, f i p represents the global features of the pth image of pedestrian category i, represents the global features of the nth image of pedestrian category j; represents the maximum Euclidean distance between the ath image of pedestrian category i and the nth image of pedestrian category i, represents the minimum Euclidean distance between the a-th image of pedestrian category i and the n-th image of pedestrian category j; function [*] + If * is greater than 0, [*] + Output *, if * is less than or equal to 0, [*] + The output is 0;
[0087] Difficult Cross-modal Triplet Loss L hct The calculation formula is as follows:
[0088]
[0089] Among them, f i cp Indicates that the pedestrian category is i and is the same as f i a Global features that do not belong to the same modality image, Indicates that the pedestrian category is j and is the same as f i a Global features that do not belong to the same modality image; represents the minimum Euclidean distance between the image of pedestrian category i and the image of pedestrian category j in another modality, maxd(f i a ,f i cp ) represents the maximum Euclidean distance between the a-th image of pedestrian category i and the image of pedestrian category i in the other modality.
[0090] Therefore, the difficult quintuple loss L HP The calculation formula is as follows:
[0091] The difficult quintuple loss L is calculated based on the global features of visible light images and infrared images.HP , and its calculation formula is as follows:
[0092]
[0093] Where N represents the number of pedestrian categories, P represents the number of visible light images or infrared images, and L hgt represents the difficult global triplet loss, L hct Representing difficult cross-modal triplet losses.
[0094] In this embodiment, the first global classifier and the second global classifier are used to classify the global features of the visible light image and the infrared image respectively, and the cross entropy loss is calculated as the global identity loss L id , and its calculation formula is as follows:
[0095]
[0096] in, represents the label of the i-th visible light image, V represents the visible light image, Represents the probability result of the global classifier classifying the i-th visible light image; is the label of the i-th infrared image, I represents the infrared image, Represents the probability result of the global classifier classifying the i-th infrared image.
[0097] In this embodiment, a visible light classifier is used to classify the local features of the visible light image, and an infrared classifier is used to classify the local features corresponding to the infrared image, and the cross entropy loss is calculated as the specific identity loss L sid , and its calculation formula is as follows:
[0098]
[0099] in, represents the probability result of the visible light classifier classifying the jth local feature of the i-th visible light image, It represents the probability result of the infrared classifier classifying the jth local feature of the i-th infrared image.
[0100] S5: Loss L based on the difficult quintuple HP , global identity loss L id and the identity-specific loss L sid The overall loss function L of the cross-modal person re-identification model is constructed, and its expression is as follows:
[0101] L=L id +L HP +λL sid
[0102] Among them, λ is the weight parameter of the loss function, and the value of λ is 0.9.
[0103] According to the overall loss function, the momentum optimizer is used to optimize the parameters of the cross-modal person re-identification model, and the momentum is set to 0.9 and the initial learning rate is 0.01. The number of model training times is set to 50, and the learning rate is decayed every 10 times, and the decay rate is 0.1. Minimize the overall loss function.
[0104] S6: Input the images in the test set and the target images to be identified into the optimized cross-modal person re-identification model, calculate the feature similarity between the target images to be identified and the images in the test set, and output the first 20 images in the test set with the highest feature similarity to the target images to be identified as the person re-identification results.
[0105] In this embodiment, the images to be retrieved in the test set and the images in the retrieval gallery are input into the optimized cross-modal pedestrian re-identification model, and the global features of the visible light images and infrared images after the first pooling layer and the second pooling layer are used to calculate the feature similarity. The first 20 images in the test set with the highest feature similarity to the target image to be identified are output as the pedestrian re-identification results.
[0106] Example 3
[0107] See also Figure 3 ,This embodiment proposes a cross-modal pedestrian re-identification system based on multi-feature learning, including a data acquisition module, a cross-modal pedestrian re-identification model and an optimization module.
[0108] During the specific implementation process, the data acquisition module randomly acquires several batches of visible light images and infrared images of pedestrians to construct training sets and test sets. In the test set, infrared images are selected as the image set to be retrieved, and visible light images are used as the retrieval library; and each batch of visible light images and infrared images in the training set are respectively input into the cross-modal pedestrian re-identification model.
[0109] The cross-modal pedestrian re-identification model includes a two-stream network, which includes a visible light modality network and an infrared modality network. The visible light image is input into the visible light modality network, and a three-dimensional tensor of the visible light image is first obtained. On the one hand, the three-dimensional tensor of the visible light image is globally averaged pooled in the visible light modality network to obtain the global features of the visible light image; on the other hand, the three-dimensional tensor of the visible light image is horizontally split in the visible light modality network to obtain several horizontal partial tensors, and the horizontal partial tensors are globally averaged pooled to obtain corresponding column vectors; the column vectors are reduced in dimension using a convolutional layer to obtain local features of the visible light image.
[0110] The infrared image is input into the infrared modality network, and the three-dimensional tensor of the infrared image is first obtained. On the one hand, the three-dimensional tensor of the infrared image is globally averaged pooled in the infrared modality network to obtain the global features of the infrared image; on the other hand, the three-dimensional tensor of the infrared image is horizontally segmented in the infrared modality network to obtain several horizontal partial tensors, and the horizontal partial tensors are globally averaged pooled to obtain corresponding column vectors; the column vectors are reduced in dimension using a convolutional layer to obtain the local features of the infrared image. According to the acquired visible light image and infrared image, the global features of the visible light image and the infrared image, as well as the local features of the visible light image and the infrared image are output.
[0111] The optimization module calculates the loss function of the cross-modal pedestrian re-identification model including the difficult quintuple loss, the global identity loss and the local feature calculation specific identity loss according to the global features of the visible light image and the infrared image, as well as the local features of the visible light image and the infrared image, and optimizes the parameters of the cross-modal pedestrian re-identification model according to the loss function.
[0112] Finally, the images to be retrieved in the test set and the images in the retrieval gallery are input into the optimized cross-modal person re-identification model. The global features of the visible light images and infrared images after the first pooling layer and the second pooling layer are used to calculate the feature similarity. The first M images in the test set with the highest feature similarity to the target images to be identified are output as the person re-identification results.
[0113] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;
[0114] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A cross-modal person re-identification method based on multi-feature learning, characterized in that: The following steps are involved: S1: Construct a cross-modal person re-identification model including a two-stream network; S2: Randomly obtain several batches of visible light images and infrared images of pedestrians to construct training sets and test sets; S3: inputting the visible light image and the infrared image into a cross-modal person re-identification model in batches, respectively, and the cross-modal person re-identification model outputs global features of the visible light image and the infrared image, and local features of the visible light image and the infrared image; S4: Calculate the difficult quintuple loss and the global identity loss according to the global features, and calculate the specific identity loss according to the local features; S5: constructing an overall loss function of the cross-modal person re-identification model according to the difficult quintuple loss, the global identity loss and the specific identity loss, and optimizing parameters of the cross-modal person re-identification model according to the overall loss function; S6: Input the images in the test set and the target images to be identified into the optimized cross-modal person re-identification model, calculate the feature similarity between the target images to be identified and the images in the test set, and output the first M images in the test set with the highest feature similarity to the target images to be identified as the person re-identification results.
2. The cross-modal person re-identification method based on multi-feature learning according to claim 1, characterized in that: In S3, the visible light image and the infrared image are respectively input into a cross-modal person re-identification model to obtain three-dimensional tensors of the visible light image and the infrared image respectively; Perform global average pooling on the three-dimensional tensors of the visible light image and the infrared image to obtain the global features of the visible light image and the infrared image; The three-dimensional tensors of the visible light image and the infrared image are horizontally divided into K horizontal partial tensors, and the K horizontal partial tensors are globally average pooled to obtain K column vectors; a convolutional layer is used to reduce the dimension of the k column vectors to obtain local features of the visible light image and the infrared image.
3. The cross-modal person re-identification method based on multi-feature learning according to claim 1, characterized in that: In S2, several batches of visible light images and infrared images of pedestrians are randomly acquired, and each batch includes P visible light images and P infrared images of N types of pedestrians, for a total of 2NP images.
4. The cross-modal person re-identification method based on multi-feature learning according to claim 3 is characterized in that: In S4, the difficult quintuple loss L is calculated based on the global features of the visible light image and the infrared image. HP , and its calculation formula is as follows: Where N represents the number of pedestrian categories, P represents the number of visible light images or infrared images, and L hgt represents the difficult global triplet loss, L hct Representation difficulty cross-modal triplet loss; The difficult global triplet loss L hgt The calculation formula is as follows: Wherein, p=1...2P, a=1...2P, p≠a, n=1...2P, i=1...N, j=1...N, j≠i, α represents the difference between the maximum Euclidean distance and the minimum Euclidean distance between images; f i a represents the global features of the a-th image of pedestrian category i, f i p represents the global features of the pth image of pedestrian category i, f j n represents the global features of the nth image of pedestrian category j; max d(f i a , f i p ) represents the maximum Euclidean distance between the a-th image of pedestrian category i and the n-th image of pedestrian category i, min d(f i a , f j n ) represents the minimum Euclidean distance between the a-th image of pedestrian category i and the n-th image of pedestrian category j; function [*] + If * is greater than 0, [*] + Output *, if * is less than or equal to 0, [*] + The output is 0; The difficult cross-modal triplet loss L hct The calculation formula is as follows: Among them, f i cp Indicates that the pedestrian category is i and is the same as f i a Global features that do not belong to the same modality image, Indicates that the pedestrian category is j and is the same as f i a Global features that do not belong to the same modality image; represents the minimum Euclidean distance between the a-th image of pedestrian category i and the image of pedestrian category j in another modality, maxd(f i a , f i cp ) represents the maximum Euclidean distance between the a-th image of pedestrian category i and the image of pedestrian category i in the other modality.
5. The cross-modal person re-identification method based on multi-feature learning according to claim 4 is characterized in that: In S4, a global classifier is used to classify the global features, and the cross entropy loss is calculated as the global identity loss L id , and its calculation formula is as follows: in, represents the label of the i-th visible light image, V represents the visible light image, Represents the probability result of the global classifier classifying the i-th visible light image; is the label of the i-th infrared image, I represents the infrared image, Represents the probability result of the global classifier classifying the i-th infrared image.
6. The cross-modal person re-identification method based on multi-feature learning according to claim 5, characterized in that: In S4, the local features of the visible light image are classified by identity using the visible light classifier, and the local features corresponding to the infrared image are classified by identity using the infrared classifier, and the cross entropy loss is calculated as the specific identity loss L sid , and its calculation formula is as follows: in, represents the probability result of the visible light classifier classifying the jth local feature of the i-th visible light image, It represents the probability result of the infrared classifier classifying the jth local feature of the i-th infrared image; According to the difficult quintuple loss L HP , global identity loss L id and the identity-specific loss L sid The overall loss function L of the cross-modal person re-identification model is constructed, and its expression is as follows: L=L id +L HP +λL sid Among them, λ is the weight parameter of the loss function.
7. The cross-modal person re-identification method based on multi-feature learning according to any one of claims 1 to 6, characterized in that: In S1, the dual-stream network includes a visible light modality network and an infrared modality network; The visible light modality network includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a first horizontal segmentation module, a first pooling layer, a visible light classifier and a first global classifier; the first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module are connected in sequence, and the output end of the fifth convolution module is connected to the input end of the first horizontal segmentation module and the first pooling layer respectively; the output end of the first horizontal segmentation module is connected to the input end of the sixth convolution module, and the output end of the sixth convolution module is connected to the input end of the visible light classifier; the output end of the first pooling layer is connected to the input end of the first global classifier; The infrared modality network includes a seventh convolution module, an eighth convolution module, a ninth convolution module, a tenth convolution module, an eleventh convolution module, a twelfth convolution module, a second horizontal segmentation module, a second pooling layer, an infrared classifier and a second global classifier; the seventh convolution module, the eighth convolution module, the ninth convolution module, the tenth convolution module and the eleventh convolution module are connected in sequence, and the output end of the eleventh convolution module is connected to the input end of the second horizontal segmentation module and the second pooling layer respectively; the output end of the second horizontal segmentation module is connected to the input end of the twelfth convolution module, and the output end of the twelfth convolution module is connected to the input end of the infrared classifier; the output end of the second pooling layer is connected to the input end of the second global classifier; The third convolution module shares parameters with the ninth convolution module; the fourth convolution module shares parameters with the tenth convolution module; the fifth convolution module shares parameters with the eleventh convolution module; the second convolution module shares parameters with the eighth convolution module; Parameters of the first global classifier and the second global classifier are shared.
8. The cross-modal person re-identification method based on multi-feature learning according to claim 7, characterized in that: The visible light modality network and the infrared modality network use BotNet as the backbone network.
9. The cross-modal person re-identification method based on multi-feature learning according to claim 7, characterized in that: The visible light classifier, the first global classifier, the infrared classifier and the second global classifier each include a fully connected layer and a normalization layer.
10. A cross-modal person re-identification system based on multi-feature learning, characterized in that: include: The data acquisition module is used to randomly acquire several batches of visible light images and infrared images of pedestrians to construct training sets and test sets; A cross-modal person re-identification model, including a two-stream network, for outputting global features of the visible light image and the infrared image, as well as local features of the visible light image and the infrared image according to the acquired visible light image and the infrared image; The similarity between the target image to be identified and the images in the test set is calculated, and the first M images in the test set with the highest feature similarity to the target image to be identified are output as the person re-identification result; The optimization module is used to calculate the loss function of the cross-modal pedestrian re-identification model including the difficult quintuple loss, the global identity loss and the local feature calculation specific identity loss according to the global features of the visible light image and the infrared image, as well as the local features of the visible light image and the infrared image, and optimize the parameters of the cross-modal pedestrian re-identification model according to the loss function.