Cross-modal pedestrian re-identification method and system based on three-modal consistency constraint
By fusing color and infrared images and employing the ternary central loss and compact intramodal constraints, the problem of large differences in intermodal and intramodal features in cross-modal pedestrian re-identification is solved, thereby improving the accuracy of pedestrian recognition under nighttime infrared images.
Patent Information
- Application Number
- CN202211239389.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-11
- Publication Date
- 2026-01-02
- Estimated Expiration
- 2042-10-11
AI Technical Summary
Existing cross-modal pedestrian re-identification methods have too large a difference between color images and infrared images, making it impossible to fully exploit modality-invariant features. This results in insufficient extraction of pedestrian information from infrared images taken at night, affecting the recognition performance.
By constructing a pedestrian recognition model based on convolutional neural networks, fusing color images and infrared images, and using image fusion technology to generate heterogeneous enhanced images, the model is optimized by reducing feature differences between and within modes through ternary central loss and compact intramodal constraints, and by using cross-entropy loss.
It effectively reduces intermodal differences, improves the accuracy of cross-modal pedestrian re-identification, and promotes pedestrian recognition accuracy under different lighting conditions.
Smart Images
Figure CN115661857B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of pattern recognition and artificial intelligence, and particularly relates to a cross-modal person re-identification method based on heterogeneous enhanced sample consistency constraint and a computing system. BACKGROUND
[0002] Person Re-Identification (Person Re-Identification) aims to search for the same person in different monitoring cameras, which has great application value in monitoring video, security and other fields. However, the general single-modal person re-identification method only studies the color image taken in the daytime, explores the robust person feature in the color image, and many existing single-modal methods have achieved performance exceeding the human eye. However, these single-modal methods cannot extract effective person information in the infrared image taken at night, so they cannot be further applied at night, which limits the application value of person re-identification. The cross-modal person re-identification method can simultaneously learn the person features of the color image taken in the daytime and the infrared image taken at night, has the value of person re-identification at night, and has important practical significance.
[0003] At present, some existing cross-modal person re-identification methods use an adversarial network to generate person images from one modality to another. Wang et al. designed an adversarial model to generate cross-modal paired person images, and then constrained the paired cross-modal images in the feature space to reduce the difference between the modalities. Choi et al. constructed an identity-preserving generative model to generate cross-modal person images with different illuminations and poses, which significantly improved the diversity of samples and promoted the accuracy of person retrieval. In addition, some works propose effective loss functions to constrain the feature distribution to simultaneously eliminate the feature differences between and within modalities. Zhu et al. proposed a heterogeneous center loss to limit the feature distribution of different modalities to learn modality-invariant features. Ye et al. proposed a dual-stream network with a ranking loss to reduce the difference between color and infrared images, which significantly improved the performance of cross-modal person re-identification. However, the method based on the generative model to generate new images is difficult to obtain effective color information from the infrared image, and the quality of the generated color image is low, which affects the performance of cross-modal person re-identification. For the existing constraint loss, it cannot consider the pixel-level difference, and it is difficult to simultaneously eliminate the differences between and within modalities, resulting in poor generalization performance. SUMMARY
[0004] The purpose of the present application is to solve the problem that the difference between color images and infrared images is too large and the modality-invariant features cannot be fully mined, and to solve the technical problem of eliminating the large difference between and within modalities. To this end, the present application provides a cross-modal person re-identification method based on heterogeneous enhanced sample consistency constraint and a computing system.
[0005] In order to achieve the object, the application provides a cross-modal pedestrian re-identification method based on heterogeneous enhanced sample consistency constraint and a computing system, which comprises the following steps:
[0006] Step S1, constructing a pedestrian identification model based on a convolutional neural network;
[0007] Step S2, fusing a color image and an infrared image based on an image fusion technology to generate a heterogeneous enhanced pedestrian image;
[0008] Step S3, inputting the color image, the infrared image and the heterogeneous enhanced image after preprocessing into the constructed pedestrian identification model to extract a feature map of the pedestrian image;
[0009] Step S4, performing a pooling operation based on the extracted feature map to obtain overall and local features of the image, inputting the features into a classifier to complete a classification task, and calculating a loss by using cross-entropy;
[0010] Step S5, constructing a triple-center loss based on the extracted overall and local features to reduce feature differences between modes;
[0011] Step S6, constructing a compact intra-modal constraint based on the extracted overall and local features to reduce feature differences within the mode;
[0012] Step S7, jointly optimizing the constructed pedestrian identification model by using the cross-entropy loss, the triple-center loss and the compact intra-modal constraint loss;
[0013] Step S8, extracting overall and local features of a pedestrian image based on the optimized pedestrian identification model, concatenating the features to perform pedestrian matching, and obtaining a pedestrian identification result.
[0014] Optionally, the step S1 comprises the following steps:
[0015] Step S11, obtaining a pre-trained ResNet-50 deep learning model and initializing parameters of the model;
[0016] Step S12, repeatedly constructing a model structure before Block3 in the ResNet-50 three times to process three different types of data respectively, and taking Block3 and Block4 in the ResNet-50 as shared modules to process the three different types of data.
[0017] Optionally, in the step S2, the existing image fusion technology FusionDN is used to fuse the color image and the infrared image to generate the heterogeneous enhanced image, wherein an image pair input into the FusionDN randomly selects one pedestrian image with the same label in the color mode and the infrared mode.
[0018] Optionally, the step S3 comprises the following steps:
[0019] Step S31, pre-processing operation is performed on the data, wherein the pre-processing of the color image and the infrared image comprises: random horizontal flipping, random cropping, random erasing are performed on the pedestrian image, and normalization operation is performed on the pedestrian image; the pre-processing of the heterogeneous enhanced image comprises: random horizontal flipping and random cropping are performed on the pedestrian image, and normalization operation is performed on the pedestrian image.
[0020] Step S32, the color image, the infrared image and the heterogeneous enhanced image after the pre-processing are input into the constructed pedestrian recognition model, and the feature maps corresponding to the color image, the infrared image and the heterogeneous enhanced image are extracted after the model.
[0021] Optionally, the step S4 comprises the following steps:
[0022] Step S41, a pooling operation is performed on the extracted color image feature map, infrared image feature map and heterogeneous enhanced image feature map, and overall features corresponding to the three kinds of images are obtained.
[0023] Step S42, the extracted color image feature map and infrared image feature map are horizontally divided into three uniform blocks, and a pooling operation is performed on each block to obtain corresponding local features.
[0024] Step S43, based on the extracted overall and local features, each feature is input into a classifier, and a prediction loss is calculated by using a cross-entropy function, wherein the loss values corresponding to the color image, the infrared image and the heterogeneous enhanced image are represented as L v ,L r ,L a :
[0025]
[0026]
[0027]
[0028] wherein t is a true label corresponding to the feature, represents the i-th feature of the color image, represents the i-th feature of the infrared image, represents the overall feature of the heterogeneous image, represents the probability that the feature is predicted as the label t, i=1, 2, 3, 4, and represents three local features and one overall feature.
[0029] Step S44, the total value of the prediction loss of the three image features calculated by using the cross-entropy function is represented as:
[0030]
[0031] where P represents the total number of pedestrian image identity categories input into the network each time, and K represents the total number of images in each pedestrian category.
[0032] Optionally, the step S5 comprises the following steps:
[0033] In step S51, based on the global and local features of the extracted color image, the constructed ternary center loss is represented as:
[0034]
[0035] where D() represents the calculation of the Euclidean distance, and p represents a hyperparameter to control the difference between different feature centers, represents the feature center of the i-th feature of the color image, represents the feature center with the same identity and the farthest distance, represents the feature center with the same identity and the farthest distance, represents the feature center with the different identity and the closest distance, represents the feature center with the different identity and the closest distance, where The calculation formula of is: K represents the total number of images of the same category, represents the i-th feature of the k-th image with identity p in the color image.
[0036] In step S52, based on the global and local features of the extracted infrared image, the constructed ternary center loss is represented as:
[0037]
[0038] where D() represents the calculation of the Euclidean distance, and p represents a hyperparameter to control the difference between different feature centers, represents the feature center of the i-th feature of the infrared image, represents the feature center with the same identity and the farthest distance, represents the feature center with the same identity and the farthest distance, represents the feature center with the different identity and the closest distance, represents the feature center with the different identity and the closest distance, where The calculation formula of is: K represents the total number of images of the same category, represents the i-th feature of the k-th image with identity p in the infrared image, + max(0, x).
[0039] In step S53, based on the global feature of the extracted heterogeneous enhanced image, the constructed ternary center loss is represented as:
[0040]
[0041] Where D() represents the calculation of Euclidean distance, and ρ represents a hyperparameter to control the difference between different feature centers. Feature centers representing the overall features of a heterogeneously enhanced image Indicates and The feature centers that share the same identity and are farthest apart Indicates and The feature centers with different identities and the closest distance. The calculation formula is: K represents the total number of images of the same category. This represents the overall features of the k-th image with identity p in the heterogeneous enhanced image.
[0042] Step S54, based on the calculated three losses, the final total loss value of the constructed ternary center loss is expressed as:
[0043]
[0044] Optionally, step S6 includes the following steps:
[0045] Step S61, based on the global and local features of the extracted color image, the constructed compact modal constraints are expressed as follows:
[0046] I vp =log(d vp +1)
[0047] in This represents the distance between the most difficult feature pairs to compute. Representation and feature center All positive samples with the same identity.
[0048] Step S62, based on the global and local features of the extracted infrared image, the constructed compact modal constraints are expressed as follows:
[0049] I rp =log(d rp +1)
[0050] in This represents the distance to the most difficult feature pair to compute. Representation and feature center All positive samples with the same identity.
[0051] Step S63, based on the overall features of the extracted heterogeneous enhanced image, the constructed compact intramodal constraint is expressed as:
[0052] I ap =log(dap +1)
[0053] wherein denotes the distance of the computed hardest feature pair, denotes the center of the feature all positive samples with the same identity.
[0054] Step S64, based on the three types of losses calculated, the final total loss value of the constructed intra-compact modal constraint is represented as:
[0055]
[0056] Optionally, in step S7, the joint cross-entropy loss, the ternary center loss and the intra-compact modal constraint loss are combined to optimize the final constraint function of the constructed pedestrian recognition model, which is represented as:
[0057] L = L ID + aL TCL + bL CIC .
[0058] wherein a and b represent hyperparameters that control their importance.
[0059] The beneficial effects of the present application are: the present application effectively fuses color images and infrared images through image fusion technology to produce heterogeneous enhanced images, which can effectively reduce the differences between modalities. In addition, the construction of heterogeneous ternary loss and intra-compact modal constraint loss implements consistency constraints on multi-modal features, sufficiently reduces the feature differences between modalities and within modalities, promotes the model to learn robust pedestrian features, and improves the accuracy of cross-modal pedestrian re-identification search. BRIEF DESCRIPTION OF DRAWINGS
[0060] Figure 1 is a flowchart of a cross-modal pedestrian re-identification method and computing system based on heterogeneous enhanced sample consistency constraints according to an embodiment of the present application;
[0061] Figure 2 is a structural framework schematic diagram of a cross-modal pedestrian re-identification method and computing system based on heterogeneous enhanced sample consistency constraints according to an embodiment of the present application. DETAILED DESCRIPTION
[0062] To make the purpose, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the specific embodiments and the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present application.
[0063] Figure 1is a flowchart of a cross-modal pedestrian re-identification method and computing system based on heterogeneous enhanced sample consistency constraints according to an embodiment of the present application, and the following will be described with an example of Figure 1 as shown in the figure. Figure 1 The cross-modal pedestrian re-identification method and computing system based on heterogeneous enhanced sample consistency constraints includes the following steps:
[0064] Step S1, based on a convolutional neural network, a pedestrian recognition model is constructed;
[0065] Further, the step S1 includes the following steps:
[0066] Step S11, a pre-trained ResNet-50 deep learning model is obtained and parameterized, and the model structure before Block3 in ResNet-50 is repeated three times to process three different types of data, and Block3 and Block4 in ResNet-50 are used as shared modules to process three different types of data.
[0067] Step S12, the model structure before Block3 in ResNet-50 is repeated three times to process three different types of data, and Block3 and Block4 in ResNet-50 are used as shared modules to process three different types of data.
[0068] Step S2, based on image fusion technology, color images and infrared images are fused to generate heterogeneous enhanced pedestrian images;
[0069] Further, in the step S2, the existing image fusion technology FusionDN is used to fuse color images and infrared images to generate heterogeneous enhanced images, wherein the input image pairs of FusionDN are randomly selected in color and infrared modalities with the same labeled pedestrian image.
[0070] In an embodiment of the present application, when image fusion is performed on the SYSU-MM01 database, a color image is first randomly selected, then a pedestrian with the same identity as the color image is randomly selected in the infrared modality, and then the selected color and infrared images are input into the FusionDN model to generate heterogeneous enhanced images.
[0071] Step S3, after the color image, the infrared image and the heterogeneous enhanced image are preprocessed, they are input into the constructed pedestrian recognition model to extract the feature map of the pedestrian image;
[0072] Further, the step S3 includes the following steps:
[0073] Step S31, pre-processing operation is performed on the data, wherein the pre-processing of the color image and the infrared image comprises: random horizontal flipping, random cropping, random erasing are performed on the pedestrian image, and normalization operation is performed on the pedestrian image; the pre-processing of the heterogeneous enhanced image comprises: random horizontal flipping and random cropping are performed on the pedestrian image, and normalization operation is performed on the pedestrian image.
[0074] Step S32, the color image, the infrared image and the heterogeneous enhanced image after pre-processing are input into the constructed pedestrian recognition model, and the feature maps corresponding to the color image, the infrared image and the heterogeneous enhanced image are extracted after the model.
[0075] Step S4, based on the extracted feature maps, a pooling operation is performed to obtain the overall and local features of the image, the features are input into a classifier to complete a classification task, and a loss is calculated by using cross-entropy;
[0076] In an embodiment of the present application, the dimension of each feature is 2048.
[0077] Further, the step S4 comprises the following steps:
[0078] Step S41, a pooling operation is performed on the extracted color image feature map, infrared image feature map and heterogeneous enhanced image feature map to obtain the overall features corresponding to the three kinds of images.
[0079] Step S42, the extracted color image feature map and infrared image feature map are horizontally divided into three uniform blocks, and a pooling operation is performed on each block to obtain the corresponding local features.
[0080] Step S43, based on the extracted overall and local features, each feature is input into a classifier, and a prediction loss is calculated by using a cross-entropy function, wherein the loss values corresponding to the color image, the infrared image and the heterogeneous enhanced image are represented as L v ,L r ,L a :
[0081]
[0082]
[0083]
[0084] wherein t is the true label corresponding to the feature, represents the i-th feature of the color image, represents the i-th feature of the infrared image, represents the overall feature of the heterogeneous image, represents that the feature The probability of predicting the label t is P (t | x) = exp (f (t | x) ) / Σ t' exp (f (t' | x) ), i = 1, 2, 3, 4, which represents three local features and one global feature.
[0085] In step S44, the total predicted loss value of the three image features calculated by the cross-entropy function is represented as:
[0086]
[0087] where P represents the total number of pedestrian image identity categories input into the network each time, and K represents the total number of images in each pedestrian category.
[0088] In an embodiment of the present application, P can be set to 8, and K can be set to 4.
[0089] In step S5, based on the extracted global and local features, a triplet center loss is constructed to reduce the feature difference between modalities.
[0090] Further, the step S5 includes the following steps:
[0091] In step S51, based on the extracted global and local features of the color image, the constructed triplet center loss is represented as:
[0092]
[0093] where D() represents the calculation of the Euclidean distance, and p represents a hyperparameter to control the distance between different feature centers. represents the feature center of the i-th feature of the color image, represents the feature center with the same identity and the farthest distance, represents the feature center with the same identity and the farthest distance, represents the feature center with the different identity and the closest distance, where represents the feature center with the different identity and the closest distance, where The calculation formula of is: K represents the total number of images of the same category, represents the i-th feature of the k-th image with identity p in the color image, [x] + = max (0, x).
[0094] In step S52, based on the extracted global and local features of the infrared image, the constructed triplet center loss is represented as:
[0095]
[0096] where D() represents the calculation of the Euclidean distance, and p represents a hyperparameter to control the distance between different feature centers. represents the feature center of the i-th feature of the infrared image, represents the feature center with the same identity and the farthest distance, the feature center with the same identity and the farthest distance, represents the feature center with the same identity and the farthest distance, the feature center with the same identity and the farthest distance, The calculation formula of is: K represents the total number of images of the same category, represents the i-th feature of the k-th image with identity p in the infrared image.
[0097] Step S53, based on the extracted overall features of the heterogeneous enhanced image, the constructed ternary center loss is represented as:
[0098]
[0099] wherein D() represents the calculation of the Euclidean distance, and p represents a hyperparameter to control the difference between different feature centers, represents the feature center of the overall feature of the heterogeneous enhanced image, represents the feature center with the same identity and the farthest distance, the feature center with the same identity and the farthest distance, represents the feature center with the same identity and the farthest distance, the feature center with the same identity and the farthest distance, The calculation formula of is: K represents the total number of images of the same category, represents the overall feature of the k-th image with identity p in the heterogeneous enhanced image.
[0100] Step S54, based on the calculated three kinds of losses, the total loss value of the finally constructed ternary center loss is represented as:
[0101]
[0102] In an embodiment of the present application, P can be set to 8.
[0103] Step S6, based on the extracted overall and local features, a compact intra-modal constraint is constructed to reduce the feature difference within the modal;
[0104] Further, the step S6 comprises the following steps:
[0105] Step S61, based on the extracted overall and local features of the color image, the constructed compact intra-modal constraint is represented as:
[0106] I vp = log(d vp +1)
[0107] wherein d represents the distance of the most difficult feature pair, represents the feature center with the same identity and the farthest distance, All positive samples with the same identity.
[0108] In step S62, based on the overall and local features of the extracted infrared image, the constructed compact intra-modal constraint is represented as:
[0109] I rp = log(d rp + 1)
[0110] wherein d represents the distance of the most difficult pair of features calculated, represents the distance to the feature center All positive samples with the same identity.
[0111] In step S63, based on the overall features of the extracted heterogeneous enhanced image, the constructed compact intra-modal constraint is represented as:
[0112] I ap = log(d ap + 1)
[0113] wherein d represents the distance of the most difficult pair of features calculated, represents the distance to the feature center All positive samples with the same identity.
[0114] In step S64, based on the three types of losses calculated, the final constructed compact intra-modal constraint total loss value is represented as:
[0115]
[0116] In step S7, the cross-entropy loss, the triple-center loss and the compact intra-modal constraint loss are jointly optimized to build the pedestrian recognition model.
[0117] Further, in step S7, the final constraint function for jointly optimizing the cross-entropy loss, the triple-center loss and the compact intra-modal constraint loss to build the pedestrian recognition model is represented as:
[0118] L = L ID + αL TCL + βL CIC .
[0119] wherein α and β represent hyperparameters for controlling their importance.
[0120] In an embodiment of the present application, α and β can be set to 0.1 and 1.0, respectively.
[0121] In step S8, based on the optimized pedestrian recognition model, the overall and local features of the pedestrian image are extracted and concatenated for pedestrian matching to obtain the pedestrian recognition result.
[0122] In an embodiment of the present application, during the testing process, the whole and local features of each image are concatenated as the last feature of each image, and the last feature of each image is 8192 dimensions.
[0123] Finally, the performance of the model based on the consistency constraint of heterogeneous enhanced samples can also be evaluated according to the correct rate of search.
[0124] Taking the large database of cross-modal pedestrian re-identification publicly available on the Internet as the test object, for example, when tested on the SYSU-MM01 database, the correct search rate of the pedestrian re-identification of the present application reaches 71.83%. The cross-modal pedestrian re-identification method of the present application effectively eliminates the feature differences between and within modalities, learns the modal invariance features, maintains the consistent distribution of features of different modalities, and greatly improves the correct search rate of cross-modal pedestrian re-identification, thus proving the effectiveness of the method of the present application.
[0125] It should be understood that the above specific embodiments of the present application are only used for illustrative or explanatory purposes of the principles of the present application, and do not constitute a limitation of the present application. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the present application shall be included in the protection scope of the present application. In addition, the appended claims of the present application are intended to cover all variations and modifications falling within the scope and boundary of the appended claims, or the equivalent forms of such scope and boundary.
Claims
1. A method for cross-modal pedestrian re-identification based on three-modal consistency constraints, characterized in that, The method comprises the following steps: Step S1, based on a convolutional neural network, a pedestrian recognition model is constructed; Step S2, based on an image fusion technology, a color image and an infrared image are fused to generate a heterogeneous enhanced pedestrian image; Step S3, after the color image, the infrared image and the heterogeneous enhanced image are preprocessed, the constructed pedestrian recognition model is inputted, and feature maps corresponding to the color image, the infrared image and the heterogeneous enhanced image are extracted; Step S4, based on the extracted feature maps, a pooling operation is performed to obtain overall and local features of the image, the features are inputted into a classifier to complete a classification task, and a loss is calculated by using cross entropy; Step S4 comprises the following steps: Step S41, a pooling operation is performed on the extracted color image feature map, infrared image feature map and heterogeneous enhanced image feature map to obtain overall features corresponding to the three images; Step S42, the extracted color image feature map and infrared image feature map are horizontally divided into three uniform blocks, and a pooling operation is performed on each block to obtain corresponding local features; Step S43, based on the extracted global and local features, input each feature into the classifier, and use the cross-entropy function to calculate the prediction loss, wherein the loss values corresponding to the color image, the infrared image and the heterogeneous enhanced image are respectively represented as , , : where t is the true label corresponding to the feature, represents the i-th feature of the color image, represents the i-th feature of the infrared image, represents the overall feature of the heterogeneous image, represents the feature predicted as label t, i = 1, 2, 3, 4, representing three local features and one overall feature; Step S44, a total value of prediction losses of the three image features calculated by using a cross entropy function is represented as: Wherein P represents a total number of pedestrian image identity categories inputted into the network each time, and K represents a total number of images in each pedestrian category; Step S5, based on the extracted overall and local features, a ternary center loss is constructed to reduce feature differences between modes; Step S6, based on the extracted overall and local features, a compact intra-modal constraint loss is constructed to reduce feature differences within modes; Step S7, the cross entropy loss, the ternary center loss and the compact intra-modal constraint loss are jointly optimized to optimize the constructed pedestrian recognition model; Step S8, based on the optimized pedestrian recognition model, overall and local features of a pedestrian image are extracted, and the features are concatenated to perform pedestrian matching to obtain a pedestrian recognition result.
2. The method of claim 1, wherein: Step S1 comprises the following steps: Step S11, a pre-trained ResNet-50 deep learning model is obtained, and parameters of the model are initialized; Step S12, a model structure before Block3 in the ResNet-50 is repeatedly constructed three times to process three different types of data respectively, and Block3 and Block4 in the ResNet-50 are taken as shared modules to process the three different types of data.
3. The method of claim 1, wherein: In step S2, an existing image fusion technology FusionDN is used to fuse the color image and the infrared image to generate the heterogeneous enhanced image, wherein image pairs inputted into the FusionDN are randomly selected to have the same labeled pedestrian image in the color mode and the infrared mode.
4. The method of claim 1, wherein: Step S3 comprises the following steps: Step S31, a preprocessing operation is performed on the data, wherein the preprocessing of the color image and the infrared image comprises: the pedestrian image is subjected to horizontal flipping, random cropping and random erasing machines, and the pedestrian image is subjected to a normalization operation; and the preprocessing of the heterogeneous enhanced image comprises: the pedestrian image is subjected to random horizontal flipping and random cropping, and the pedestrian image is subjected to a normalization operation; Step S32, input the color image, infrared image and heterogeneous enhanced image after preprocessing into the constructed pedestrian recognition model, and extract the feature maps corresponding to the color image, infrared image and heterogeneous enhanced image.
5. The method of claim 1, wherein: The step S5 comprises the following steps: Step S51, based on the global and local features of the extracted color image, the constructed ternary center loss is represented as: where D() denotes the Euclidean distance, denotes the hyper-parameter to control the gap between different feature centers, denotes the feature center of the i-th feature of the color image, denotes the feature center of the i-th feature of the color image, denotes the feature center with the same identity and the farthest distance, denotes the feature center with the same identity and the farthest distance, denotes the feature center with the different identity and the closest distance, where The calculation formula of is: K denotes the total number of images of the same class, denotes the i-th feature of the k-th image with identity p in the color image, + = max(0, x); Step S52, based on the global and local features of the extracted infrared image, the constructed ternary center loss is represented as: where D() denotes the Euclidean distance, denotes the hyper-parameter to control the gap between different feature centers, denotes the feature center of the i-th feature of the infrared image, denotes the feature center of the i-th feature of the infrared image with the same identity, denotes the feature center with the same identity and the farthest distance, denotes the feature center with the same identity and the farthest distance, denotes the feature center with the different identity and the nearest distance, where The calculation formula of is as follows: K denotes the total number of images of the same category, denotes the i-th feature of the k-th image with the identity p in the infrared image; Step S53, based on the global feature of the extracted heterogeneous enhanced image, the constructed ternary center loss is represented as: Where D() represents the calculation of Euclidean distance, This represents a hyperparameter used to control the difference between different feature centers. Feature centers representing the overall features of a heterogeneously enhanced image Indicates and The feature centers that share the same identity and are farthest apart Indicates and The feature centers with different identities and the closest distance. The calculation formula is: K represents the total number of images of the same category. This represents the overall features of the k-th image with identity p in the heterogeneous enhanced image; Step S54, based on the calculated three types of loss, the total loss value of the finally constructed ternary center loss is represented as: 。 6. The method of claim 5, wherein: Step S6 comprises the following steps: Step S61, based on the global and local features of the extracted color image, the constructed compact intra-modal constraint is represented as: wherein denotes the distance of the most difficult feature pair calculated, denotes the distance of the most difficult feature pair calculated, all positive samples with the same identity; Step S62, based on the global and local features of the extracted infrared image, the constructed compact intra-modal constraint is represented as: wherein denotes the distance of the hardest feature pair computed, denotes the distance of the hardest feature pair computed, all positive samples with the same identity; Step S63, based on the global feature of the extracted heterogeneous enhanced image, the constructed compact intra-modal constraint is represented as: wherein denotes the distance of the hardest feature pair computed, denotes the distance of the hardest feature pair computed, all positive samples with the same identity; Step S64, based on the calculated three types of loss, the total loss value of the finally constructed compact intra-modal constraint is represented as: 。 7. The method of claim 6, wherein: In step S7, the cross-entropy loss, the ternary center loss and the compact intra-modal constraint loss are combined to jointly optimize the final constraint function of the constructed pedestrian recognition model, which is represented as: wherein and denotes a hyperparameter controlling its importance.
8. A cross-modality pedestrian re-identification system based on three-modality consistency constraints, characterized in that, The system comprises the following modules: A model construction module for constructing a pedestrian recognition model based on a convolutional neural network; A fusion module for fusing a color image and an infrared image based on an image fusion technology to generate a heterogeneous enhanced pedestrian image; A feature map extraction module for inputting the color image, infrared image and heterogeneous enhanced image after preprocessing into the constructed pedestrian recognition model to extract the feature maps corresponding to the color image, infrared image and heterogeneous enhanced image; A cross-entropy loss calculation module for performing a pooling operation to obtain global and local features of an image based on the extracted feature maps, inputting these features into a classifier to complete a classification task, and calculating a loss using cross-entropy; comprising the following steps: Step S41, performing a pooling operation on the extracted color image feature map, infrared image feature map and heterogeneous enhanced image feature map to obtain global features corresponding to the three types of images; Step S42, dividing the extracted color image feature map and infrared image feature map into three uniform blocks horizontally, and performing a pooling operation on each block to obtain corresponding local features; Step S43, based on the extracted global and local features, input each feature into the classifier, and use the cross-entropy function to calculate the prediction loss, wherein the loss values corresponding to the color image, the infrared image and the heterogeneous enhanced image are respectively represented as , , : where t is the true label corresponding to the feature, represents the i-th feature of the color image, represents the i-th feature of the infrared image, represents the overall feature of the heterogeneous image, represents the feature predicted as label t, i = 1, 2, 3, 4, representing three local features and one overall feature; Step S44, the total value of the predicted loss of the three types of image features calculated using the cross-entropy function is represented as: Where P represents the total number of pedestrian image identity categories input into the network each time, and K represents the total number of images in each pedestrian category; A ternary center loss calculation module for constructing a ternary center loss based on the extracted global and local features to reduce the feature difference between modalities; An intra-modal constraint loss calculation module for constructing a compact intra-modal constraint loss based on the extracted global and local features to reduce the feature difference within modalities; An optimization module for jointly optimizing the cross-entropy loss, ternary center loss and compact intra-modal constraint loss to jointly optimize the constructed pedestrian recognition model; The recognition module is configured to extract overall and local features of the pedestrian image based on the optimized pedestrian recognition model, concatenate the features, and perform pedestrian matching to obtain a pedestrian recognition result.