A cross-modal person re-identification method based on overall and partial constraints
Through the hybrid cross-section dual-path feature learning network and overall constraint loss function, the problem of large differences between modes in cross-modal pedestrian re-identification is solved, and the recognition accuracy is improved, especially in the case of insufficient light.
Patent Information
- Application Number
- CN202210124910.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-02-10
AI Technical Summary
The existing cross-modal pedestrian re-identification method has a large difference between visible light and infrared modes and has fewer data sets, resulting in poor recognition accuracy, especially in the case of insufficient light.
A hybrid crossover dual-path feature learning network (HCDFL) is proposed. Through the overall constraint and partial triple-center loss function, combined with random horizontal flip and random erase data enhancement, local pedestrian features are extracted and differences between modes are narrowed. Modal-specific identity loss and cross-entropy loss are adopted to improve identification performance.
Effectively reduce the differences between infrared modes and RGB modes, improve pedestrian characteristics characterization capabilities, and improve recognition accuracy, especially in the case of insufficient light.
Smart Images

Figure CN114495281B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of pedestrian re-identification, and in particular relates to a cross-modal pedestrian re-identification method based on overall and partial constraints. Background Art
[0002] Person re-identification is a specific pedestrian retrieval task that uses computer vision technology to determine whether a specific pedestrian exists in an image or video. In recent years, with the continuous development of society, people have paid more and more attention to public safety issues, and pedestrian re-identification has aroused great research interest. At present, most studies mainly process human images taken by visible light cameras. However, these methods have many limitations. For example, many crimes occur at night, and traditional visible cameras cannot capture clear images. Therefore, these methods are not effective in low light conditions.
[0003] Most of the existing research focuses on person re-identification in the visual modality. Compared with visible images, infrared images lack rich color information, so the commonly used person re-identification methods based on visible light images are not feasible in infrared images. After searching the existing technical literature, it was found that Wu et al. proposed a large-scale cross-modal person re-identification dataset called SYSY-MM01. At the same time, they evaluated three commonly used neural network structures: single stream, double stream and asymmetric fully connected layer, and proposed deep zero padding for training single stream network. Zheng et al. introduced an end-to-end coupled pedestrian re-identification learning and data generation joint learning framework to solve the infrared-visible light pedestrian re-identification problem. Mang et al. proposed a dynamic dual attention aggregation learning framework to avoid the model learning being easily disturbed by noise and becoming unstable. Li et al. added an X modality to the network to solve the modality difference. Many existing methods focus on narrowing the difference between infrared and visual modalities. When clues are shared, the recognition accuracy is not ideal. The above methods have solved the cross-modal pedestrian re-identification problem to a certain extent, but there are still shortcomings.
[0004] Therefore, there are several urgent problems to be solved in cross-modal person re-identification: (1) There is a large difference between the visible mode and the infrared mode. Although the existing methods have achieved certain results, there is still a lot of room for improvement; (2) There are few cross-modal pedestrian re-identification datasets, resulting in insufficient training data. This is not only a problem in cross-modal pedestrian re-identification, but also a common problem in pedestrian re-identification. The academic community does not have complex and large-scale datasets to study, and the industry has a large amount of data but cannot be open sourced due to privacy issues. Summary of the invention
[0005] To solve the above problems, the present invention proposes a hybrid cross dual-path feature learning network (HCDFL), which deeply extracts local pedestrian features from two different modalities; utilizes a novel overall constraint and partial triplet-center loss function to improve the inter-class and intra-class differences from different modalities and the same modality, respectively, so as to better characterize the local features of pedestrians and improve the overall recognition performance; at the same time, random horizontal flipping and random erasing are used to enhance the data to expand the training data.
[0006] The present invention provides a cross-modal person re-identification method based on overall and partial constraints, comprising the following steps:
[0007] S1. Use two independent and identical branch networks to extract pedestrian information features in different modes from RGB images and infrared images of the same scene;
[0008] S2, the extracted features are evenly divided into P horizontal components from top to bottom, the P horizontal components are projected into the common space, and the joint representation of the modality-specific features and the modality-common features is output;
[0009] S3. Construct a multi-loss function, which includes modality-specific identity loss, cross entropy loss, and the proposed overall constraint and partial triplet-center loss. Use the multi-loss function to mix and cross the joint features, and use the modality distance constraint to narrow the image difference between the infrared modality and the RGB modality to obtain the best recognition performance.
[0010] Furthermore, the multi-loss function is:
[0011]
[0012] Among them and They represent the specific modality softmax identity loss of the RGB branch and the infrared branch, respectively. CE represents the cross entropy loss, L WCPTL represents the overall constraint and the partial triplet-center loss function; λ represents the pre-training coefficient used to balance the overall loss function.
[0013] Furthermore, the overall constraint process in the loss function includes two steps: first, the distance between different pedestrian samples in the same modality is enlarged, and the distance between the same pedestrian samples in the RGB modality and the infrared modality is reduced; then, the distance between the same pedestrian samples in the two modalities is further reduced, the similarity of pedestrian identity is improved, and the difference between different pedestrian samples in the modality is reduced; given the deep features of pedestrians in different modalities, Where 1≤i≤N, and Respectively represent the identity of the i-th pedestrian in RGB and infrared modes, and denote the pth and qth identities of the i-th pedestrian in RGB and infrared modes respectively. The specific formula of the overall constraint LW is as follows:
[0014]
[0015] The part-center triplet loss formula is as follows:
[0016]
[0017] Among them, LP represents the partial triplet-center loss, x i and z i Represents RGB image features and infrared image features respectively, c 1yi and c 2yi Respectively represent the i-th category center in RGB and infrared modalities, y i represents the identity label of the i-th sample, α represents the bias, N represents the batch size, represents the Euclidean distance, [x] + =max(0,x);
[0018] In summary, the overall constraint and partial triplet-center loss function can be expressed as:
[0019] L WCPTL =L W +L P .
[0020] Furthermore, modality-specific identity loss: Since the pedestrian features in RGB images and infrared images are very different, different networks are used to obtain feature representations in different modalities. Softmax loss is used to predict the pedestrian identity in each modality, and its formula can be expressed as:
[0021]
[0022]
[0023] In the formula and Respectively indicate belonging to and The i-th RGB image features and infrared image features of the class, and Respectively represent the weight W in the previous fully connected layer V and W I The jth column of b V and b I They represent the bias of RGB and infrared modes respectively, M represents the total number of pedestrians, and N V and N IRespectively represent the number of RGB image and infrared image training samples in the same batch, and denote the identity loss functions for RGB images and infrared images, respectively.
[0024] Furthermore, in order to make the feature representations of the same person in different modes similar, the cross entropy loss function is introduced as follows:
[0025]
[0026] Among them, y i Represents the true label of the i-th input image, that is, the p component features of each input image share the label information of the image.
[0027] The beneficial effects described in the present invention are as follows: a cross-modal pedestrian re-identification method based on overall and partial constraints described in the present invention proposes a hybrid cross dual-path feature learning network, which designs a modality shared parameter layer and a modality unique parameter layer to extract features from pedestrian images of different modalities; secondly, the network horizontally cuts pedestrian features to better learn local and global features of the image and improve the pedestrian feature representation capability; at the same time, for the model embedding layer, the network cross-combines features to form multiple different batch combinations, which is beneficial to feature matching and modality distance constraints; when designing the loss function, the consistency constraints of feature distribution within different modality data classes and the inter-class correlation constraint criteria are fully considered, and a novel overall constraint and partial triplet-center loss function is proposed to improve modality differences so that samples of the same category are closer to the category center and away from other category centers; when training data, random horizontal flipping and random erasing are used to enhance data to expand the training data. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 It is a schematic diagram of the process of the method of the present invention;
[0029] Figure 2 The end-to-end framework diagram of the cross-modal person re-identification based on the overall constraint and the partial triplet-center loss according to the present invention is shown;
[0030] Figure 3 Schematic diagram of the principle of combining triplet loss, center loss and softmax loss respectively;
[0031] Figure 4 It is a schematic diagram of the overall constraint of the present invention;
[0032] Figure 5 It is a schematic diagram of the partial-triplet center loss of the present invention;
[0033] Figure 6This is a schematic diagram of the recognition effect of the present invention on the SYSU-MM01 and RegDB datasets. DETAILED DESCRIPTION
[0034] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.
[0035] Combination Figure 1 and Figure 2 The method proposed in the present invention first uses the RGB branch and infrared branch of ResNet50, both of which are backbone networks, to extract pedestrian information under different modalities respectively, and uses the average pooling layer to evenly divide the extracted features into p horizontal components from top to bottom; then, the horizontal cut features are projected into the common space, and the joint representation of the modality-specific features and the modality-common features is output; finally, the modality-specific identity loss, cross entropy loss, and the proposed overall constraint and partial triple-center loss are used to mix and cross the joint features, and the optimal recognition performance is obtained through the modality distance constraint.
[0036] The overall constraint and partial triplet-center loss proposed in the present invention firstly constrain the distance between different modalities as a whole, thereby reducing the difference between RGB and infrared modalities; secondly, the loss function combines the triplet loss and the center loss to learn the centers of the RGB modality and the infrared modality respectively, so that samples of the same category are closer to the category center and away from the centers of other categories, thereby improving the intra-modality difference.
[0037] 1. Hybrid Cross Dual Path Feature Learning Network
[0038] In pedestrian re-identification based on visual single modality, the commonly used method is to horizontally segment the pedestrian image, extract local features, and then perform feature matching on the pedestrian image. However, in infrared-visible light pedestrian re-identification, the image taken by the infrared camera is very different from the visible image; the infrared image only retains inherent features, such as the overall appearance and posture information of the pedestrian, but loses very important information such as color and lighting. Therefore, the commonly used method cannot be used to solve the infrared-visible light pedestrian re-identification problem.
[0039] Combination Figure 2As shown in the figure, this paper adopts a dual-stream structure as the basic structure. The main reason is that the single-stream structure uses a common feature extraction network, and it is impossible to accurately extract the features of RGB and infrared images in this process; in addition, since the single-stream structure shares global parameters, it leads to serious neglect of the local features of pedestrians. In the dual-stream structure, the shallow network parameters are individually for each modality, while the deep network parameters are shared, which takes into account both local features and global features, improving the recognition performance. Therefore, the present invention adopts a traditional two-way local feature network, which consists of two parts: a feature extractor and a feature embedding.
[0040] The infrared-visible light person re-identification dataset can be expressed as D = {V, I}, where V represents the RGB image and I represents the infrared image.
[0041] In the feature extraction stage, the corresponding pedestrian features are obtained after feature extraction by the backbone network ResNet50, and the network with the last average pooling layer and its subsequent structures is removed, so as to achieve the purpose of expanding the receiving domain area and enriching the feature granularity. In particular, both branches use the same network structure. This design will make the high-level feature output more expressive of high-level semantics and make the identity discrimination ability of the features stronger.
[0042] In the feature embedding stage, pedestrian features are first divided horizontally into p identical components to learn the low-dimensional embedding space between two heterogeneous modalities; then, a global pooling layer is used on each component to obtain p 2048-dimensional features. In order to further reduce the feature dimension, a 1×1 convolution layer is used for each 2048-dimensional component feature to reduce the dimension, and finally a 256-dimensional feature expression is obtained;
[0043] At the same time, in order to avoid gradient vanishing and calculate internal covariate offset, a batch normalization layer is added after each fully connected layer; finally, the shared layer is used as a projection function to project the features of two different modalities into a common embedding space to bridge the differences between the two modalities.
[0044] In the training phase, the network model is trained by combining modality-specific identity loss, cross entropy loss, and the proposed overall constraint and partial triple-center loss to improve the recognition accuracy. The joint representation features of the RGB branch and the infrared branch are divided into three groups using hybrid cross-training, namely partial constraint, overall constraint, and cross entropy loss, where the partial constraint and overall constraint constitute the proposed overall constraint and partial triple-center loss function. In the testing phase, the features of the detection image and the gallery image are extracted separately, and then the high-dimensional image features are connected to form the final feature descriptor.
[0045] The traditional dual-path feature learning network extracts pedestrian features separately through the backbone network, and then directly outputs the features by fusing them through the weight sharing module, attempting to learn cross-modal information directly from the two original modalities. Relevant experimental results show that these methods are not enough to narrow the gap between the two modalities. The network proposed in the present invention cross-combines pedestrian features to form multiple different batch combinations and cooperates with multiple loss functions. This feature cross-combination method is conducive to balancing the model's expression and learning capabilities for the unique features and shared features of different modal data, and effectively improving the matching ability between multimodal data.
[0046] The modality-specific loss function here directly utilizes the modality information and retains the most original pedestrian features; the cross entropy loss is used to identify the identity of pedestrians and extract RGB and infrared modality features and combine them into a batch; within the same batch, the features of RGB images and infrared images are consistent, so paired batches are constructed using partial constraints and overall constraints respectively.
[0047] The multi-loss function is constructed by joint collaboration as shown in formula (1), including modality-specific identity loss, cross entropy loss, global constraint loss and partial triple-center loss; the total loss function of the proposed framework can be expressed as:
[0048]
[0049] in, and They represent the specific modality softmax identity loss of the RGB branch and the infrared branch, respectively. ce Denotes the cross entropy loss L WCPT represents the overall constraint and the partial triplet-center loss function. λ represents the pre-training coefficient used to balance the overall loss function.
[0050] Modality-specific identity loss: Since the pedestrian features in RGB images and infrared images are very different, different networks are used to obtain feature representations in different modalities. Softmax loss is used to predict the pedestrian identity in each modality, and its formula can be expressed as:
[0051]
[0052] in, and Respectively indicate belonging to and The i-th RGB image features and infrared image features of the class, and Respectively represent the weight W in the previous fully connected layer V and W I The jth column of b Vand b I They represent the bias of RGB and infrared modes respectively, M represents the total number of pedestrians, and N V and N I Respectively represent the number of RGB image and infrared image training samples in the same batch.
[0053] Cross entropy loss: In order to make the feature representations of the same person in different modes similar, the cross entropy loss function is introduced as follows:
[0054]
[0055] That is, the p component features of each input image share the label information of the image.
[0056] 2. Global Constraint and Partial Triplet-Center Loss
[0057] The present invention proposes a novel overall constraint and partial triplet-center loss, which improves the inter-class and intra-class differences from different modalities and the same modality respectively, thereby improving the overall recognition performance.
[0058] The triplet loss function is often used in fields such as face recognition and pedestrian re-identification. This loss function not only has the characteristic of shortening the intra-class distance, but also has the characteristic of increasing the inter-class distance. For the infrared-visible light pedestrian re-identification task, pedestrian images not only have inter-class distances in the same modality, but also inter-class distances in different modalities.
[0059] The triplet loss function formula is as follows:
[0060]
[0061] in, Represent the feature representations of anchor points, positive sample images, and negative sample images respectively, and The identity information is the same as and The identity information is different, α represents the bias, N represents the batch size, represents the Euclidean distance, [x] + =max(0,x).
[0062] Combination Figure 3 (a) It can be seen that although the combination of these two loss functions achieves good results, it will lead to uneven data distribution and unstable model performance. Center loss was first applied to the field of face recognition. Center loss is used to constrain the distance between the sample and its category center and learn a center for each individual. The formula of the center loss function is as follows:
[0063]
[0064] Among them, x i is the feature representation, y i corresponds to x i Category, c yi Represents category y i The center of M represents the minimum batch size. represents the Euclidean distance.
[0065] Combination Figure 3 (b) It can be seen that the key to learning inter-modal features with the overall constraint loss is to reduce the cross-modal difference. Due to the drastic visual changes, the cross-modal difference may be huge, which will greatly reduce the pedestrian re-identification performance, so it is necessary to reduce the cross-modal difference as a whole.
[0066] Combination Figure 4 The overall constraint process in the loss function proposed by the present invention includes two steps: first, the distance between different pedestrian samples in the same modality is enlarged, while the distance between the same pedestrian samples in the RGB modality and the infrared modality is reduced; then, the distance between the same pedestrian samples in the two modalities is further reduced, so as to improve the similarity of pedestrian identity and reduce the difference between different pedestrian samples in the modality. Given the deep features of pedestrians in different modalities Where 1≤i≤N, and Respectively represent the identity of the i-th pedestrian in RGB and infrared modes, and They represent the p-th and q-th identities of the i-th pedestrian in RGB and infrared modes respectively. The specific formulas are as follows:
[0067]
[0068] Combination Figure 5 ,By combining the two loss functions, samples under two modalities can be considered at the same time, which is conducive to reducing the intra-class differences under the same modality, as well as narrowing the differences between modalities and improving the recognition accuracy.
[0069] The part-center triplet loss formula is as follows:
[0070]
[0071] Among them, x i and z i Represents RGB image features and infrared image features respectively, c 1yi and c 2yi Respectively represent the i-th category center in RGB and infrared modalities, y i represents the identity label of the i-th sample, α represents the bias, N represents the batch size, represents the Euclidean distance, [x] +=max(0,x).
[0072] In summary, the overall constraint and partial triple-center loss function can be expressed as:
[0073] L WCPTL =L W +L P (8)
[0074] Figure 6 The results shown are schematic diagrams of the recognition effect of the method of the present invention on the SYSU-MM01 and RegDB data sets. It can be seen from the figure that the recognition effect of the proposed method is relatively ideal and the recognition accuracy is high.
[0075] In summary, the present invention aims at the problems existing in cross-modal pedestrian re-identification. On the one hand, a hybrid cross dual-path feature learning network (HCDFL) is proposed, which deeply extracts local pedestrian features from two different modalities. The network model first extracts pedestrian features under different modalities, then horizontally cuts the extracted features into p components and maps them to the common space, which is used to horizontally cut the local features and global features of the image to improve the pedestrian feature representation ability; finally, the modality-specific identity loss, cross entropy loss and the proposed loss function work together to improve the overall performance; on the other hand, the present invention proposes a novel overall constraint and partial triple-center loss, which improves the inter-class and intra-class differences from different modalities and the same modality respectively, and is used to better characterize the local features of pedestrians and improve the overall recognition performance. The loss function proposed in the present invention first uses the overall constraint to reduce the difference between different modalities; then, by fusing the triple loss and the center loss, it is used to expand the difference between different categories in the same modality, so that the samples of the same category are closer to their center and away from the centers of other categories. In addition, due to the limited amount of image data, random horizontal flipping and random erasing are used to enhance the data in training to expand the training data. Experimental results on two public datasets, SYSU-MM01 and RegDB, show that the proposed method achieves excellent recognition performance.
[0076] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.
Claims
1. A cross-modal person re-identification method based on overall and partial constraints, characterized in that: The steps of the method are: S1. Use two independent and identical branch networks to extract pedestrian information features in different modes from RGB images and infrared images of the same scene; S2, the extracted features are evenly divided into P horizontal components from top to bottom, the P horizontal components are projected into the common space, and the joint representation of the modality-specific features and the modality-common features is output; S3. Construct a multi-loss function, use the multi-loss function to mix and cross the joint features, and reduce the image difference between the infrared mode and the RGB mode through the modal distance constraint to obtain the best recognition performance; Wherein, the multi-loss function is: Among them and They represent the specific modality softmax identity loss of the RGB branch and the infrared branch, respectively. CE represents the cross entropy loss, L WCPTL represents the overall constraint and the partial triplet-center loss function; λ represents the pre-training coefficient used to balance the overall loss function; The overall constraint process includes two steps: first, the distance between different pedestrian samples in the same modality is enlarged, while the distance between the same pedestrian samples in the RGB modality and the infrared modality is reduced; then, the distance between the same pedestrian samples in the two modalities is further reduced to improve the similarity of pedestrian identity and reduce the difference between different pedestrian samples in the modality; given the deep features of pedestrians in different modalities, Where 1≤i≤N, and Respectively represent the identity of the i-th pedestrian in RGB and infrared modes, and denote the pth and qth identities of the i-th pedestrian in RGB and infrared modalities respectively, and the overall constraint L W The specific formula is as follows: The part-center triplet loss formula is as follows: Among them, L P represents the partial triplet-center loss, x i and z i Represents RGB image features and infrared image features respectively, c 1yi and c 2yi Respectively represent the i-th category center in RGB and infrared modalities, y i represents the identity label of the i-th sample, α represents the bias, N represents the batch size, represents the Euclidean distance, [x] + =max(0,x); In summary, the overall constraint and partial triplet-center loss function can be expressed as: L WCPTL =L W +L P .
2. The cross-modal person re-identification method based on overall and partial constraints according to claim 1, characterized in that: Modality-specific identity loss: Since the pedestrian features in RGB images and infrared images are very different, different networks are used to obtain feature representations in different modalities. Softmax loss is used to predict the pedestrian identity in each modality. The formula can be expressed as: In the formula and Respectively indicate belonging to and The i-th RGB image features and infrared image features of the class, and Respectively represent the weight W in the previous fully connected layer V and W I The jth column of b V and b I They represent the bias of RGB and infrared modes respectively, M represents the total number of pedestrians, and N V and N I Respectively represent the number of RGB image and infrared image training samples in the same batch, and denote the identity loss functions for RGB images and infrared images, respectively.
3. The cross-modal person re-identification method based on overall and partial constraints according to claim 1, characterized in that: The cross entropy loss function is: Among them, y i Represents the true label of the i-th input image, that is, the p component features of each input image share the label information of the image.
Citation Information
Patent Citations
Cross-modal pedestrian re-identification method based on difficult quintuple
CN111597876A
Cross-modal pedestrian re-identification method based on local information learning
CN112434796A
Cross-modal pedestrian re-identification method based on sample center loss function
CN113569639A
Method for person re-identification based on deep model with multi-loss fusion training strategy
US20200285896A1
Adversarial cross-media retrieving method based on restricted text space
WO2019148898A1
