Cross-modal person re-identification method based on parameter sharing and feature learning of intermediate modalities

Abstract

A cross-modal person re-identification method based on parameter sharing and feature learning of intermediate modalities. An intermediate modality generator generates intermediate modal images that mitigate modal differences. The original image and the intermediate modal image are then fed into a self-designed two-stream backbone network for feature extraction. Global and local feature learning is performed on the extracted features to reduce modal differences at both coarse and fine granularity levels. A joint loss function is used to select identity-distinguishing features from shared modalities and optimize the network model. This method has achieved promising results in cross-modal person re-identification tasks.

Description

Technical Field

[0001] The present application belongs to the field of deep learning technology, and in particular relates to a cross-modal person re-identification method based on parameter sharing and feature learning of intermediate modalities. Background Art

[0002] Person re-identification can be viewed as an image retrieval task, which aims to match a single pedestrian image in a query set with pedestrian images in a gallery set taken by disjoint cameras. Due to its important role in intelligent video surveillance and public safety, it has attracted increasing attention in the field of computer vision and has become one of the hottest research areas in academia.

[0003] Visible-Visible Re-ID (VV-ReID) is the most common unimodal person re-ID task, where both the query and gallery sets are images captured by visible cameras. With the advancement of computer vision, high performance has been achieved. However, in real-world applications, such as intelligent video surveillance, which requires 24 / 7 operation, cameras that can automatically switch from visible light to infrared light are widely used. In this case, the query image is acquired from an infrared camera at night, and the gallery image is captured by a visible camera during the day, which is consistent with the cross-modal Visible-Infrared (VI Re-ID) task.

[0004] There are two main problems in the VI Re-ID task. First, the different wavelengths of visible light and thermal imaging cameras lead to large cross-modal differences. Second, there are intra-modal differences caused by posture, perspective, occlusion, lighting, etc.

[0005] To mitigate modality discrepancies in VI Re-ID tasks, there are two mainstream approaches. For example, some traditional methods attempt to find a unified feature space where the discrepancy between the two modalities is minimized. These methods extract modality-invariant features by training an end-to-end network, typically employing a two-stream network consisting of two independent feature extraction networks and a shared feature embedding network that projects the features into a common feature space. Another approach, based on generative adversarial networks, converts images from one modality to another, thereby mitigating modality discrepancies. However, this approach suffers from complex network structures, difficulty converging, and the generated images still exhibit modality discrepancies from the real images. Summary of the Invention

[0006] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a cross-modal pedestrian re-identification method based on parameter sharing and feature learning of intermediate modalities. It continues to use the intermediate modality generator of the prior art to generate intermediate modality images from natural light and infrared images, and independently designs a new parameter-sharing two-stream network for feature extraction. At the same time, it independently designs global and local feature learning modules, which overcomes the problem of cross-modal retrieval of images in different modalities and different postures.

[0007] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0008] The cross-modal person re-identification method based on parameter sharing and feature learning of intermediate modalities includes the following steps:

[0009] S1. Obtain a training dataset, where each training sample is a visible light and infrared image with an identity tag. Input the training sample into the proposed intermediate modality generator, and the generated intermediate modality image has the same identity tag as the original image.

[0010] S2. Generate the intermediate modal image M vis , M ir Together with the original image VIS and IR, they are fed into the designed parameter-sharing two-stream network to extract modality-shared features;

[0011] S3. The extracted feature maps are fed into the designed global and local feature learning modules to reduce modality differences at both coarse-grained and fine-grained levels.

[0012] S4. Use L dcl loss trains the intermediate modality generator so that the generated intermediate modality images are closer to L id loss and L hc_tri The combination of loss supervises local features, L cc The loss supervises the global features. The overall loss L is the sum of the above losses. The parameters of the intermediate mode generator and the two-stream backbone network are optimized through the back propagation of the overall loss L.

[0013] S5. Input the image to be identified into the trained network, extract the features of the image to be identified and the features of the database images respectively, and complete the identification of the image to be identified by comparing the image features.

[0014] The intermediate modal generator in step S1 includes two modal information encoders E whose parameters are not shared. vis , E ir , and a parameter-sharing modality information decoder D projects VIS and IR images into a unified feature space.

[0015] The parameter-shared dual-stream network designed in step S2 divides the resnet50 network, where the parameters of the zeroth convolution layer and the first convolution layer are not shared and are used to construct a modality-specific feature extraction network to extract modality-specific features; the parameters of the second, third, and fourth convolution layers are shared and are used to construct a modality-shared feature embedding network to extract modality-shared features.

[0016] The global and local feature learning in step S3 is to obtain local features by horizontally slicing pedestrians and to use L id loss and L hc_tri loss combined supervision; L is used for global features cc loss supervision.

[0017] The overall loss L in step S4 is expressed as follows:

[0018] L=L id +0.5L dcl +L hc_tri +L cc (1)

[0019] Among them L id is the sliding label cross entropy loss, L hc_tri Heterogeneous centric triplet loss, L id and L hc_tri Jointly supervise local features, L dcl is the distribution consistency loss, which is used to make the images generated by the intermediate modality generator close to each other; L cc is the center aggregation loss, which is used to supervise the global features.

[0020] Among them, L id is defined as:

[0021]

[0022] Where C is the number of categories in the training set, y represents the ID label of the person, and p i is the predicted probability of class i, and ε is a small constant that encourages the model to be less confident on the training set.

[0023] L dcl The calculation formula is as follows:

[0024]

[0025] Where N is the number of batches in the training phase. vis and M ir The total number of images, f() is the output before the fully connected layer of the network, mean[AB] is the mean of the difference between A and B, L dclThe optimization of will make the two types of M-mode images generated by the intermediate modality generator MMG more similar.

[0026] The heterocentric triplet loss between the four modal images is:

[0027] L hc_tri =L hc_tri (V,I)+L hc_tri (V,M ir )+L hc_tri (I,M vis )+L hc_tri (M vis ,M ir ) (4)

[0028] Among them L hc_tri (V,I) represents the heterocentric triplet loss for VIS and IR images, and its specific formula is expressed as:

[0029]

[0030] Where P represents the random selection of P identities in each iteration, where represents the VIS, IR, and feature center of the anchor sample for the i-th identity in the batch. ||AB||2 is the Euclidean distance between A and B, and ρ is a hyperparameter. The calculation formula defined in formula (4) for images from other modalities is the same as that in formula (5).

[0031] The total center aggregation loss is:

[0032] L cc =L cc (V,I)+L cc (V,M vis )+L cc (M vis ,I)+L cc (M vis ,M ir ) (6)

[0033] Among them L cc (V,I) represents the center aggregation loss of VIS and IR images, and its specific formula is expressed as:

[0034]

[0035] in is the current batch with label y iThe average value of the features of , P is the number of identities in the current batch, and σ is the minimum margin between centers. Where B represents the number of input images of each modality, 2B is the number of images after V and I are batched together, and the calculation formula definition between images of other modalities in formula (6) is the same as formula (7).

[0036] Furthermore, σ is set to 0.3, ρ is set to 0.3, and ε is set to 0.1.

[0037] This method uses two encoders with independent nonlinear parameters to encode visible light and infrared images. A weight-sharing decoder then projects the images into a unified common space, resulting in a uniform intermediate modality. This significantly mitigates modal differences between visible light and infrared images. During the feature extraction phase, parameter sharing within the two-stream network is fully explored. Furthermore, global and local feature learning modules are designed to mitigate modal differences at both coarse and fine granularity levels. This model achieves superior performance compared to other state-of-the-art methods.

[0038] The advantages of the present invention are: a lightweight nonlinear intermediate modality generator (MMG) is adopted, and the generated M (intermediate) modality image and the original image are fed into the backbone network to reduce the modality difference. Secondly, a new two-stream network (TSN) is designed, and a modality-specific feature extraction network and a modality-shared feature embedding network are constructed by splitting the resnet-50 model, and the influence of parameter sharing in the two-stream network on cross-modal image feature extraction is explored. In addition, a global and local feature learning module (GPLFL) is designed, and global information can provide complete state-invariant feature information, and local information can provide more fine-grained features. In the framework of global feature learning, the center clustering loss is used to guide identity learning, and in the framework of local feature learning, we use hetero-center triplet loss to alleviate the strict constraints of triplet loss. Extensive experiments have been carried out on two challenging SYSU-MM01 and RegDB datasets to verify that the method of the present invention is superior to the most advanced methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the method of the present invention;

[0040] Figure 2 It is a network schematic diagram of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0042] In one embodiment, Figure 1 As shown in the figure, a cross-modal person re-identification method based on parameter sharing and feature learning of intermediate modalities is proposed, including:

[0043] Step S1: Obtain a training data set, where each training sample in the training data set is a first image and a second image with an identity label, where the first image and the second image are respectively one of a natural light image and an infrared image, and input the training sample into an intermediate modality generator to generate an intermediate modality image with the same identity label.

[0044] The present invention uses the dataset SYSU-MM01 as a training dataset, which consists of infrared images and natural light images with identity annotations. Infrared images and natural light images with the same identity ID are used as a training sample.

[0045] The training samples are fed into the Middle Modality Generator (MMG). In a specific embodiment, Figure 2 As shown in , the input of MMG is an image pair of different modalities with the same identity. Let Represents the paired VIS and IR images in the dataset. Where I is a set of image pairs. 3×H×W corresponds to the channel, height, and width respectively. All input images are resized to 3×384×192. Let As the intermediate modality image M generated from the VIS and IR images vis and M ir The MMG consists of two modal information encoders E whose parameters are not shared. vis , E ir , and a parameter-sharing modality information decoder D projects VIS and IR images into a unified feature space.

[0046] The formula for encoding the VIS image is:

[0047] I Vtc =E vis (I vis ) (8)

[0048] The formula for encoding the IR image is:

[0049] I Itc =E ir (I ir ) (9)

[0050] Since the modal difference between VIS and IR mainly comes from this channel, E vis ,E irFirst, a 3×1 fully connected layer is used to transform the three-channel image I vis and I ir Encoded as a single-channel image Since the relationship between VIS and IR images is highly nonlinear, a RELU activation layer is used to increase the nonlinear capability of the network. Finally, two 1×1 fully connected layers are used to further adjust I c .

[0051] The decoder D adopts parameter sharing to transform the single channel image I c Decoded into a three-channel image The formula is:

[0052]

[0053]

[0054] The proposed decoder D consists of a 1×3 fully connected layer at the channel level, followed by a ReLU activation layer to obtain a three-channel intermediate modality map I m The generated M-modality images have the same labels as the VIS and IR images.

[0055] Step S2: M-Modality, VIS and IR images are input into the backbone network together to assist the VI-ReID task in reducing modality differences.

[0056] like Figure 2 In the embodiment shown, the VIS and IR images are passed through the intermediate modality generator to generate M vis and M ir The image, also called M-modal image, is then input into a two-stream network together with the VIS and IR images in the training samples to extract image features.

[0057] In a specific embodiment, the dual-stream network is constructed using Resnet-50, including the zeroth convolution layer, the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer, wherein the parameters of the zeroth convolution layer and the first convolution layer are not shared and are used to construct a modality-specific feature extraction network to extract modality-specific features; the parameters of the second, third, and fourth convolution layers are shared and are used to construct a modality-shared feature embedding network to extract modality-shared features.

[0058] like Figure 2 As shown, the feature extraction network of the visible light branch is represented as a function ψ V ,Infrared branch feature extraction network is represented by function ψ I , to learn modality-specific information; the feature embedding network is represented as a function ψ VI, which is used to project modality-specific features into a unified feature space to learn shared features. Given the light image and infrared image, they can be represented as VIS and IR, and input into the MMG network to obtain the intermediate modality image M vis and M ir , we will VIS, IR, M vis and M ir The concatenation is performed on batches and fed into the two-branch network, thus forming a batch size of 4B, where B represents the number of input images for each modality.

[0059] The 3D features output by the network can be expressed as: F∈{V,V m ,I,IR m}.in:

[0060] V=ψ VI (ψ V (VIS)

[0061] V m =ψ VI (ψ V (M vis ))

[0062] I=ψ VI (ψ I (IR)

[0063] IR m =ψ VI (ψ I (M ir ))(11)

[0064] Where V, I are the visible light and infrared features output by the network, V m , IR m is the intermediate modal feature output by the network, such as Figure 2 As shown, for simplicity, we only use one output to represent the data of these four modalities.

[0065] Step S3: The extracted feature maps are fed into the designed global and local feature learning modules to reduce modality differences at both coarse-grained and fine-grained levels.

[0066] Given a pedestrian image, it is represented as a 3D feature map after the features are extracted by the two-stream network, such as Figure 2 As shown, the following operations are performed on the 3D feature map:

[0067] Local feature learning:

[0068] 1) Divide the 3D feature map into P parts evenly in the horizontal direction to obtain local features of the human body, such as Figure 2 As shown, P=4.

[0069] 2) For each local feature map X = C × H × W, use global average pooling to C × 1 × 1, and then change the shape to a 1D feature vector.

[0070] 3) First, the heterocentric triplet loss (L hc_tri ), then normalize the 1D local features and send them to the fully connected layer to perform the classification task and calculate the classification loss L id There are P local features that require P normalization layers and P classifiers, and they are all parameter-unshared.

[0071] Global feature learning:

[0072] 1) The 3D feature map is not segmented to retain global information. Global average pooling (avgpool) is used and then compressed into a 1D feature vector.

[0073] 2) Using the center aggregation loss function L cc To constrain the 1D feature vector, the purpose of the center clustering loss is to cluster the features to their centers, so that the distribution of different modal features under the same ID is more concentrated.

[0074] By learning local and global features, the model can reduce modal differences at both coarse-grained and fine-grained levels.

[0075] Step S4: Use L dcl Train the intermediate modality generator, L id With L hc_tri The combined supervised local features, L cc Supervise global features and optimize the parameters of the intermediate modal generator and the two-stream backbone network through back propagation of loss.

[0076] In order to further narrow the gap between the M-modal images generated by VIS and IR images, the distribution consistency loss (L dcl ). L dcl The calculation formula is as follows:

[0077]

[0078] Where N is the number of batches in the training phase. vis and M ir The total number of images, f() is the output before the fully connected layer of the network, mean[AB] is the mean of the difference between A and B, L dcl The optimization of will make the two types of M-modal images generated by the generator MMG more similar.

[0079] Use sliding label cross entropy loss L for local features idand heterocentric triplet loss L hc_tri For calculation, label smoothing cross entropy loss is often used in classification tasks, which can prevent the model from overfitting. The label smoothing cross entropy loss formula is defined as:

[0080]

[0081] Where C is the number of categories in the training set, y represents the ID label of the person, and p i is the predicted probability of category i, and ε is a small constant that encourages the model not to be too confident in the training set. In this paper, ε is set to 0.1.

[0082] Then, using hetero-center triplet loss (L hc_tri ) replaces the triplet loss, which can relax the strict constraints of triplet loss and replace the comparison between the anchor sample and all other samples with the comparison between the center of the anchor sample and the center of all other samples. Since the input of the network is M-modal images and original VIS and IR images, where M-modal images are used to assist network training, a batch of size 4B is formed, where B represents the number of input images of each modality. B = P × K means that there are P random identities, K photos are selected under each identity, and the first B is set to I vis , the second B is The third B is I ir , the fourth B is And calculate the heterocentric triplet loss for them.

[0083] First, in a mini-batch, the feature centers of each identity of the four modalities are calculated:

[0084]

[0085] in represents the features of the jth visible light image of the i-th person, k represents K randomly selected images under each identity, and the feature definitions of the other three modalities are similar.

[0086] The heterocentric triplet loss for VIS and IR images can be defined as:

[0087]

[0088] Where P represents the random selection of P identities in each iteration, ||AB||2 is the Euclidean distance between A and B, and ρ is a hyperparameter, which is set to 0.3 in this paper.

[0089] Calculate the loss for other modalities, and the total heterocentric triplet loss is:

[0090] Lhc_tri =L hc_tri (V,I)+L hc_tri (V,M ir )+L hc_tri (I,M vis )+L hc_tri (M vis ,M ir ) (4)

[0091] In addition, in order to learn the relationship between identities and aggregate features to their centers, the extracted modal features are kept close to each other. The center aggregation loss is calculated for the global features output by the network, where the center aggregation loss formula for V and I is:

[0092]

[0093] in is the current batch with label y i The average value of the features of , P is the number of identities in the current batch, σ is the minimum margin between centers, and σ is set to 0.3. Where B represents the number of input images of each modality, and 2B is the number of images after V and I are batch-joined. The center aggregation loss between other modalities is similar to formula (7). The final cross-modal center aggregation loss is calculated as:

[0094] L cc =L cc (V,I)+L cc (V,M vis )+L cc (M vis ,I)+L cc (M vis ,M ir ) (6)

[0095] The proposed network can be optimized in an end-to-end manner by minimizing the overall loss L, which is composed of several losses:

[0096] L=L id +0.5L dcl +L hc_tri +L cc (1)

[0097] Step S5: Input the image to be identified into the trained model to extract query and gallery image features, and calculate the similarity to obtain the re-identification result.

[0098] The specific implementation method is as follows: the image to be identified is input into the trained network to extract features, the extracted features are saved, and the cosine similarity (Cosine Similarity) is calculated between the query image features and the gallery image features for matching. The images to be identified are sorted from largest to smallest according to the cosine similarity to obtain the re-identification results.

[0099] The formula for calculating cosine similarity is as follows:

[0100]

[0101] Where A and B are the IR image features in the query and the VIS image features in the gallery, respectively, represented as n-dimensional vectors. · represents the vector inner product, and |||| represents the modulus of the vector. Cosine similarity measures the similarity between two vectors. A larger cosine similarity indicates a closer match between the features.

[0102] The above-described embodiments merely represent several implementation methods of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A cross-modal person re-identification method based on parameter sharing and feature learning of intermediate modalities, characterized by: The following steps are involved: S1. Obtain a training dataset, where each training sample is a visible light and infrared image with an identity tag. Input the training sample into the proposed intermediate modality generator, and the generated intermediate modality image has the same identity tag as the original image. S2. Generate the intermediate modal image M vis , M ir Together with the original image VIS and IR, they are fed into the designed parameter-sharing two-stream network to extract modality-shared features; S3. The extracted feature maps are fed into the designed global and local feature learning modules to reduce modality differences at both coarse-grained and fine-grained levels. S4. Use L dcl loss trains the intermediate modality generator so that the generated intermediate modality images are closer to L id loss and L hc_tri The combination of loss supervises local features, L cc loss supervises the global features. The overall loss L is the sum of the above losses. The parameters of the intermediate modal generator and the two-stream backbone network are optimized through the back propagation of the overall loss L. The overall loss L is expressed as follows: L=L id +0.5L dcl +L hc_tri +L cc (1) Among them L id is the sliding label cross entropy loss, L hc_tri Heterogeneous centric triplet loss, L id and L hc_tri Jointly supervise local features, L dcl is the distribution consistency loss, which is used to make the images generated by the intermediate modality generator close to each other; L cc is the center aggregation loss, which is used to supervise the global features; Central aggregation loss L cc Expressed as: L cc =L cc (V,I)+L cc (V,M vis )+L cc (M vis ,I)+L cc (M vis ,M ir ) (6) Among them L cc (V,I) represents the center aggregation loss of VIS and IR images, and its specific formula is expressed as: in is the current batch with label y i The average value of the features, P is the number of identities in the current batch, and σ is the minimum margin between centers; where B represents the number of input images of each modality, and 2B is the number of images after batch splicing of VIS images and IR images; the center aggregation loss between other modalities is similar to formula (7); S5. Input the image to be identified into the trained network, extract the features of the image to be identified and the features of the database images respectively, and complete the identification of the image to be identified by comparing the image features.

2. The cross-modal person re-identification method based on parameter sharing and feature learning of intermediate modalities according to claim 1 is characterized in that: The intermediate modal generator in step S1 includes two modal information encoders E whose parameters are not shared. vis , E ir , and a parameter-sharing modality information decoder D projects VIS and IR images into a unified feature space.

3. The cross-modal person re-identification method based on parameter sharing and feature learning of intermediate modalities according to claim 1 is characterized in that: The parameter-sharing two-stream network designed in step S2 is used to divide the resnet50 network, where the parameters of the zeroth convolution layer and the first convolution layer are not shared to construct a modality-specific feature extraction network to extract modality-specific features; The parameters of the second, third, and fourth convolutional layers are shared and used to construct a modality-shared feature embedding network to extract modality-shared features.

4. The cross-modal person re-identification method based on parameter sharing and feature learning of intermediate modalities according to claim 1 is characterized in that: The global and local feature learning in step S3 is to obtain local features by horizontally slicing pedestrians and to use L id loss and L hc_tri loss combined supervision; L is used for global features cc loss supervision.

5. The cross-modal person re-identification method based on parameter sharing and feature learning of intermediate modalities according to claim 1 is characterized in that: Step S4 L id is defined as: Where C is the number of categories in the training set, y represents the ID label of the person, and p i is the predicted probability of category i, and ε is a small constant that encourages the model not to be too confident in the training set; L dcl The calculation formula is as follows: Where N is the number of batches in the training phase. vis and M ir The total number of images, f() is the output before the fully connected layer of the network, mean[AB] is the mean of the difference between A and B, L dcl The optimization of will make the two types of M-mode images generated by the intermediate mode generator MMG more similar; The heterocentric triplet loss between the four modal images is: L hc_tri =L hc_tri (V,I)+L hc_tri (V,M ir )+L hc_tri (I,M vis )+L hc_tri (M vis ,M ir ) (4) Among them L hc_tri (V,I) represents the heterocentric triplet loss for VIS and IR images, and its specific formula is expressed as: Where P represents the random selection of P identities in each iteration, They represent the feature centers of the VIS image, IR image, and anchor sample of the i-th identity in the batch respectively; ||AB||2 is the Euclidean distance between A and B, and ρ is a hyperparameter; the calculation formulas for other modalities are the same as the above formulas.

6. The cross-modal person re-identification method based on parameter sharing and feature learning of intermediate modalities according to claim 5 is characterized in that ,σ is set to 0.3,ρ is set to 0.3, andε is set to 0.1.

Images