A dual-channel person re-identification method based on attention twin network

By using the combination of attention twin network and two-way long and short-term memory network in pedestrian recognition, semantic information in pedestrian recognition method is extracted, and the problem of insufficient recognition accuracy and generalization ability in the prior art is solved, and cross-border Effective identification of modal data sets.

CN117132937BActive Publication Date: 2024-12-06NORTHEASTERN UNIV FOSHAN GRADUATE SCHOOL OF INNOVATION +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311140621.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-06
Publication Date
2024-12-06
Estimated Expiration
2043-09-06

AI Technical Summary

Technical Problem

The existing pedestrian re-identification method fails to effectively utilize the semantic information between the query set and the gallery set, and has poor generalization ability and cannot solve the cross-modal problem, resulting in insufficient recognition accuracy and generalization ability.

Method used

Using the dual-channel pedestrian re-identification method based on attention twin network, the samples and gallery sets are learned through the dual-channel network combined with the attention mechanism, the semantic information between the sample pictures and gallery pictures is extracted, and the two-way long and short-term periods are used. Memory network calculates the semantic features of the upper and lower levels to improve recognition accuracy and generalization ability.

Benefits of technology

It has achieved improvement in accuracy and generalization ability of pedestrian re-identification, can effectively process cross-modal data sets, and improved recognition accuracy and the degree of attention of the model to important information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117132937B_ABST
    Figure CN117132937B_ABST
Patent Text Reader

Abstract

The present invention provides a dual-channel pedestrian re-identification method based on an attention twin network, comprising: sending a library image and a sample image to be identified into a first twin network for significant feature extraction, and obtaining a shallow feature vector of each image; performing similarity measurement on the shallow feature vectors of the sample image and the library image, calculating the loss L1 of the first twin network, and obtaining k library images with the highest similarity according to the loss value; sending the k library images and the sample image into a second twin network to calculate their loss L2; ​​at the same time, inputting the sample image and the k library images into a bidirectional long short-term memory network in order of similarity to calculate the upper and lower semantic feature vectors and the mean square error loss L3, and the upper and lower semantic feature vectors are aggregated to obtain a picture sorting feature vector; fusing the loss L2 of the second twin network with the picture sorting feature vector and calculating the distance, and then performing similarity aggregation to output a list of pictures that are most similar to the sample image to be identified.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian re-identification and relates to a dual-channel pedestrian re-identification method based on an attention twin network. Background Art

[0002] Person re-identification is an important task in the field of computer vision, which aims to accurately identify and track the same pedestrian in different surveillance cameras. Person re-identification technology is widely used in video surveillance, crowd management, intelligent transportation and other fields. It can improve social security, convenience and intelligence, and bring positive impacts to people's lives and social development. However, most of the current person re-identification methods independently calculate the feature representation of each person image in the query set and the gallery set, making the features of the query set and the gallery set independent of each other. This method does not utilize the potential semantic information between the query set and the gallery set, and most of them adopt a post-processing strategy, which is independent of model training and does not jointly optimize the features of person re-identification and contextual semantic information to obtain better compatibility. At the same time, the existing methods of person re-identification technology have poor generalization ability and can only achieve good results in individual similar data sets. They can only solve single-modal problems and cannot solve cross-modal problems. Summary of the invention

[0003] In order to overcome the limitations of cross-modality and recognition accuracy, the present invention proposes a dual-channel pedestrian re-identification method based on an attention twin network. This method uses a dual-channel network combining a twin network and an attention mechanism to learn samples and gallery sets, and achieves good recognition effects by extracting semantic information between sample images and gallery images.

[0004] The present invention provides a dual-channel pedestrian re-identification method based on an attention twin network, comprising:

[0005] Step 1: Send the gallery images and sample images to be identified into the first twin network for significant feature extraction to obtain the shallow feature vector of each image;

[0006] Step 2: Measure the similarity of the shallow feature vectors of the sample image and the library image, calculate the loss L1 of the first twin network, and obtain the k library images with the highest similarity based on the loss value;

[0007] Step 3: Send k gallery images and sample images to the second twin network to calculate their loss L2;

[0008] Step 4: Input the sample image and k gallery images into the bidirectional long short-term memory network in the order of similarity to calculate the upper and lower semantic feature vectors and the mean square error loss L3. The upper and lower semantic feature vectors are aggregated to obtain the image sorting feature vector;

[0009] Step 5: Fusion the loss L2 of the second twin network with the image sorting feature vector and calculate the distance, and then perform similarity aggregation to output a list of images that are most similar to the sample image to be identified.

[0010] The dual-channel pedestrian re-identification method based on the attention twin network of the present invention has at least the following beneficial effects:

[0011] 1. The pedestrian re-identification method of the present invention adopts a dual-channel network model that combines a twin network with an attention mechanism. The model can fully extract the contextual semantic features and multi-scale information between samples and galleries to achieve good recognition results.

[0012] 2. The feature extraction module in the second twin network of the present invention integrates the feature pyramid and the channel attention module, which can fully explore the similarities between sample images and library images.

[0013] 3. The pedestrian re-identification method of the present invention can improve the performance and generalization ability of the task by learning the intrinsic structure and correlation of the data and improving the model's attention to important information; the twin network used has the advantages of shared feature extraction, learning comparable feature representation, and powerful comparison performance, which can effectively improve the accuracy of re-identification.

[0014] 4. The pedestrian re-identification method of the present invention novelly proposes a dual-channel branch to extract the one-to-one correspondence information and the upper and lower sequence information between the sample image and the library image in parallel, and also has good learning ability for cross-modal data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the overall network framework diagram of the present invention;

[0016] Figure 2 It is a flow chart of a dual-channel pedestrian re-identification method based on an attention twin network of the present invention;

[0017] Figure 3 is a structural diagram of the attention feature pyramid network of the present invention;

[0018] Figure 4 It is a structural diagram of the feature pyramid module of the present invention. DETAILED DESCRIPTION

[0019] Figure 1This is the overall network framework diagram of the present invention. The present invention adopts a dual-channel pedestrian re-identification method based on the attention twin network. The library pictures and sample pictures are input into the first twin network for significant feature extraction, and the loss L1 of the first twin network is calculated based on the combined distance. Then the similarity is measured to obtain k library pictures with the highest similarity to the sample. Then the sample and the obtained k library pictures are input into the second twin network to obtain its loss L2. At the same time, the sample picture is combined with the library picture and input into the bidirectional long short-term memory network to calculate the upper and lower semantic feature vectors and the mean square error loss L3, and the picture sorting feature vector is obtained after aggregation. The pedestrian re-identification result is obtained by combining the loss L2 of the second twin network with the picture sorting feature vector.

[0020] like Figure 2 As shown, a dual-channel pedestrian re-identification method based on an attention twin network of the present invention includes:

[0021] Step 1: Send the gallery images and the sample images to be identified into the first twin network for significant feature extraction to obtain the shallow feature vector of each image. The specific steps of step 1 are:

[0022] Step 1.1: Randomly select M person identities in the gallery to construct each training batch, and uniformly sample N images of each person identity, so that there are M×N images in each training batch;

[0023] Step 1.2: Use the first twin network with ResNet-101 as the feature extraction module to extract significant features from the images in the training batch, and obtain the shallow feature vector A of the sample image and the shallow feature vector of each image. Where i=1,2,…M×N.

[0024] In specific implementation, the ResNet-101 includes a convolutional layer, four-stage residual blocks, a global average pooling layer and a fully connected layer; each residual block contains multiple convolutional layers, a batch normalization layer, a ReLu activation function and a skip link.

[0025] Step 2: perform similarity measurement on the shallow feature vectors of the sample image and the library image, calculate the loss L1 of the first twin network, and obtain the k library images with the highest similarity according to the loss value. Step 2 is specifically as follows:

[0026] Step 2.1: Use the combined distance of Euclidean distance and Manhattan distance to calculate the loss of the first twin network, and then measure the similarity between the sample image and the gallery image. Specifically, the loss L1 of the first twin network is calculated according to the following formula:

[0027]

[0028]

[0029]

[0030] Among them, D e is the Euclidean distance, D m is the Manhattan distance, A is the shallow feature vector of the sample image, is the shallow feature vector of the i-th gallery image, n is the number of images in each training mini-batch, that is, n = M × N;

[0031] Step 2.2: Sort the images in the gallery according to the calculated loss values ​​to obtain the k images with the smallest loss values, which are the k gallery images with the highest similarity.

[0032] Step 3: Send k gallery images and sample images to the second twin network to calculate their loss L2, specifically:

[0033] Step 3.1: Build a second twin network including the attention feature pyramid network and the loss calculation module;

[0034] Step 3.2: Send k gallery images and sample images into the attention feature pyramid network to obtain the deep feature vector of each image;

[0035] like Figure 3 As shown in the figure, the attention feature pyramid network includes: sampling layer, 4 feature pyramid modules, feature fusion module, channel attention module and convolution layer; k gallery images and sample images are sent to the attention feature pyramid network and the following operations are performed:

[0036] Step 3.2.1: After the image is sent to the attention feature pyramid network, it is scaled through the sampling layer and the number of channels of each branch image is reduced to one-fourth of the original number of channels. The images with sizes of 0.75, 1, 1.25, and 1.5 times the original size are recorded as f1~f4 and sent to the corresponding feature pyramid modules respectively;

[0037] Step 3.2.2: If Figure 4 As shown in Figure 2, images f1 to f4 undergo three N×1 and 1×N convolutions in the corresponding feature pyramid modules, and the results of the three convolutions are concatenated to obtain the local feature out f1 ~out f4 ; The values ​​of N in the four feature pyramid modules are 3, 5, 7, and 9 respectively;

[0038] Step 3.2.3: Local features out output by the feature pyramid module f1 ~out f4 Perform feature fusion as shown in the following formula to obtain fused local features F1~F4, and then concatenate F1~F4 to obtain global features;

[0039] F1=out f1

[0040] F2=F1+out f2

[0041] F3=F2+out f3

[0042] F4=F3+out f4

[0043] Step 3.2.4: The global features are summed pixel by pixel with the input image to obtain an image with fused global features, and then convolution is performed to output the deep feature vector of each image;

[0044] Step 3.2.5: The image with global features is sent to the channel attention module, and passes through the global average pooling layer, fully connected layer, ReLU layer, fully connected layer, and Sigmoid layer to obtain the second channel attention weight. The second channel attention weight is fed back to the sampling layer to guide the sampling operation.

[0045] Step 3.3: Use the channel attention mechanism to calculate the first channel attention weight of the deep feature vector, and feed the first channel attention weight back to the ResNet-101 module of the first twin network to guide it to extract the shallow feature vector;

[0046] Step 3.4: Use Contrastive Loss as the loss function and calculate the loss L2 of the sample image and the k gallery images based on the deep feature vector;

[0047] In specific implementation, the loss L2 is calculated according to the following formula:

[0048]

[0049]

[0050] Among them, X1 is the deep feature vector of the sample image, is the deep feature of the jth dimension of the sample image; X m is the deep feature vector of the gallery image, is the j-th dimension of the deep features of the gallery image; P represents the feature dimension; D m is the Euclidean distance between the deep feature vectors of the sample image and the library image; Y represents the label of whether the sample image and the library image match, and Q is the set threshold;

[0051] When Y=1, that is, the sample image is similar to the gallery image, the loss function formula is:

[0052]

[0053] That is, when the sample image and the gallery image are similar, if the Euclidean distance in the feature space is large, it means that the current model is not good, so the loss is increased;

[0054] When Y=0, that is, the sample image and the gallery image are not similar, the loss function formula is:

[0055]

[0056] That is, when the sample image and the library image are not similar, if the Euclidean distance in their feature space is small, the loss value will become larger.

[0057] Step 4: Input the sample image and k gallery images into the bidirectional long short-term memory network in order of similarity to calculate the upper and lower semantic feature vectors and the mean square error loss L3. The upper and lower semantic feature vectors are aggregated to obtain the image sorting feature vector. The specific step 4 is as follows:

[0058] Step 4.1: Sort the sample image and k gallery images in order of similarity and input them into the bidirectional long short-term memory network. When t>1, calculate the contextual semantic feature vector according to the following formula:

[0059]

[0060]

[0061]

[0062] When t=1, the feature vector between pictures is calculated according to the following formula:

[0063] h1=F Lf (W1A+W2)

[0064] h1'=F Lf (W3A+W4h'2)

[0065]

[0066] in, is the shallow feature vector of the t-th image sorted by similarity, is the tth contextual semantic feature vector; h t is the current forward hidden layer state vector, h t-1 is the previous forward hidden layer state vector, h t ' is the current reverse hidden layer state vector, h t ' +1is the next reverse hidden layer state vector; W1~W6 are the shared weights of each input unit, which are continuously optimized during the network training process to adapt to the pedestrian re-identification task; The first contextual semantic feature vector, h1 is the first forward hidden layer state vector, and h1' is the first reverse hidden layer state vector;

[0067] F Lf is the hidden layer equation, and the hyperbolic tangent function is used as the activation function:

[0068]

[0069] Among them, x is the input, that is, and

[0070] F Lo is the output layer equation, using the ReLU function as the activation function:

[0071]

[0072] Among them, x is the input, that is, W5h t +W6h t ', α=0.03.

[0073] Step 4.2: Calculate the mean square error loss L3 of the long short-term memory network according to the following formula:

[0074]

[0075] Step 4.3: Use two fully connected layers based on the attention mechanism to generate a one-dimensional scalar for each image feature vector, and then use the softmax function F S Generate aggregate attention weights, and calculate the image sorting feature vector based on the aggregate attention weights:

[0076]

[0077]

[0078] Among them, W7 and W8 are the weights of the fully connected layer, b7 and b8 are the corresponding biases, and γ is the aggregate attention weight.

[0079] Step 5: The loss L2 of the second twin network is integrated with the image sorting feature vector and the distance is calculated. Then, similarity aggregation is performed to output a list of images that are most similar to the sample image to be identified. Step 5 is specifically as follows:

[0080] Step 5.1: Add the loss L2 output by the second twin network to each image sorting feature vector element by element to obtain the corresponding fused feature vector;

[0081] Step 5.2: Use L2 norm to normalize multiple fused feature vectors;

[0082] Step 5.3: Calculate the distance between the normalized fusion features by using the cosine similarity method for the normalized fusion feature vector to obtain the fusion feature similarity matrix;

[0083] Step 5.4: Perform weighted averaging on the fusion feature similarity matrix to obtain the aggregated similarity of each fusion feature;

[0084] Step 5.5: Sort the images corresponding to the fusion features according to the similarity after aggregation, and output the sorted image list.

[0085] In specific implementation, the overall loss L of the first twin network, the second twin network and the long short-term memory network is calculated by the following formula: A :

[0086] L A =L1+L2+L3

[0087] By calculating L A Update the weights of the first twin network, the second twin network and the long short-term memory network to minimize the loss function and optimize the model parameters to achieve the purpose of pedestrian re-identification.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the concept of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A dual-channel pedestrian re-identification method based on attention twin network, characterized in that: include: Step 1: Send the gallery images and sample images to be identified into the first twin network for significant feature extraction to obtain the shallow feature vector of each image; Step 2: Measure the similarity of the shallow feature vectors of the sample image and the library image, calculate the loss L1 of the first twin network, and obtain the k library images with the highest similarity based on the loss value; Step 3: Send k gallery images and sample images to the second twin network to calculate their loss L2; Step 4: Input the sample image and k gallery images into the bidirectional long short-term memory network in the order of similarity to calculate the upper and lower semantic feature vectors and the mean square error loss L3. The upper and lower semantic feature vectors are aggregated to obtain the image sorting feature vector; Step 5: Fusion the loss L2 of the second twin network with the image sorting feature vector and calculate the distance, and then perform similarity aggregation to output a list of images that are most similar to the sample image to be identified.

2. The dual-channel person re-identification method based on the attention twin network as claimed in claim 1, characterized in that: The step 1 is specifically as follows: Step 1.1: Randomly select M person identities in the gallery to construct each training batch, and uniformly sample N images of each person identity, so that there are M×N images in each training batch; Step 1.2: Use the first twin network with ResNet-101 as the feature extraction module to extract significant features from the images in the training batch, and obtain the shallow feature vector A of the sample image and the shallow feature vector v of each gallery image. i a , where i=1,2,…M×N.

3. The dual-channel person re-identification method based on the attention twin network as claimed in claim 2, characterized in that: The ResNet-101 includes a convolutional layer, four-stage residual blocks, a global average pooling layer and a fully connected layer; each residual block contains multiple convolutional layers, a batch normalization layer, a ReLu activation function and a skip link.

4. The dual-channel person re-identification method based on the attention twin network as claimed in claim 2, characterized in that: The step 2 is specifically as follows: Step 2.1: Use the combined distance of Euclidean distance and Manhattan distance to calculate the loss of the first twin network, and then measure the similarity between the sample image and the gallery image. Specifically, the loss L1 of the first twin network is calculated according to the following formula: Among them, D e is the Euclidean distance, D m is the Manhattan distance, A is the shallow feature vector of the sample image, is the shallow feature vector of the i-th gallery image, n is the number of images in each training mini-batch, that is, n = M × N; Step 2.2: Sort the images in the gallery according to the calculated loss values ​​to obtain the k images with the smallest loss values, which are the k gallery images with the highest similarity.

5. The dual-channel person re-identification method based on the attention twin network as claimed in claim 2, characterized in that: In step 3, k gallery images and sample images are sent to the second twin network to calculate their losses, specifically: Step 3.1: Build a second twin network including the attention feature pyramid network and the loss calculation module; Step 3.2: Send k gallery images and sample images into the attention feature pyramid network to obtain the deep feature vector of each image; Step 3.3: Use the channel attention mechanism to calculate the first channel attention weight of the deep feature vector, and feed the first channel attention weight back to the ResNet-101 module of the first twin network to guide it to extract the shallow feature vector; Step 3.4: Use Contrastive Loss as the loss function and calculate the loss L2 of the sample image and the k gallery images based on the deep feature vector.

6. The dual-channel person re-identification method based on the attention twin network as claimed in claim 5, characterized in that: The attention feature pyramid network includes: a sampling layer, four feature pyramid modules, a feature fusion module, a channel attention module and a convolution layer; k gallery images and sample images are sent to the attention feature pyramid network and the following operations are performed: Step 3.2.1: After the image is sent to the attention feature pyramid network, it is scaled through the sampling layer and the number of channels of each branch image is reduced to one-fourth of the original number of channels. The images with sizes of 0.75, 1, 1.25, and 1.5 times the original size are recorded as f1~f4 and sent to the corresponding feature pyramid modules respectively; Step 3.2.2: Images f1~f4 undergo three N×1 and 1×N convolutions in the corresponding feature pyramid modules, and the results of the three convolutions are concatenated to obtain the local feature out f1 ~out f4 ; The values ​​of N in the four feature pyramid modules are 3, 5, 7, and 9 respectively; Step 3.2.3: Local features out output by the feature pyramid module f1 ~out f4 Perform feature fusion as shown in the following formula to obtain fused local features F1~F4, and then concatenate F1~F4 to obtain global features; F1=out f1 F2=F1+out f2 <h2 style=";text-align:left;direction:ltr">F3=F2+out<h2 style=";text-align:left;direction:ltr"> f3 F4=F3+out f4 Step 3.2.4: The global features are summed pixel by pixel with the input image to obtain an image with fused global features, and then convolution is performed to output the deep feature vector of each image; Step 3.2.5: The image with global features is sent to the channel attention module, and passes through the global average pooling layer, fully connected layer, ReLU layer, fully connected layer, and Sigmoid layer to obtain the second channel attention weight. The second channel attention weight is fed back to the sampling layer to guide the sampling operation.

7. The dual-channel person re-identification method based on the attention twin network as claimed in claim 5, characterized in that: In step 3.4, the loss L2 is calculated according to the following formula: Among them, X1 is the deep feature vector of the sample image, is the deep feature of the jth dimension of the sample image; X m is the deep feature vector of the gallery image, is the j-th dimension of the deep features of the gallery image; P represents the feature dimension; D m is the Euclidean distance between the deep feature vectors of the sample image and the library image; Y represents the label of whether the sample image and the library image match, and Q is the set threshold; When Y=1, that is, the sample image is similar to the gallery image, the loss function formula is: When Y=0, that is, the sample image and the gallery image are not similar, the loss function formula is:

8. The dual-channel person re-identification method based on the attention twin network as claimed in claim 1, characterized in that: The step 4 is specifically as follows: Step 4.1: Sort the sample image and k gallery images in order of similarity and input them into the bidirectional long short-term memory network. When t>1, calculate the contextual semantic feature vector according to the following formula: When t=1, the feature vector between pictures is calculated according to the following formula: h1=F Lf (W1A+W2) in, is the shallow feature vector of the t-th image sorted by similarity, is the tth contextual semantic feature vector; h t is the current forward hidden layer state vector, h t-1 is the previous forward hidden layer state vector, h t ' is the current reverse hidden layer state vector, h t ' +1 is the next reverse hidden layer state vector; W1~W6 are the shared weights of each input unit, which are continuously optimized during the network training process to adapt to the pedestrian re-identification task; v1 o The first contextual semantic feature vector, h1 is the first forward hidden layer state vector, and h1' is the first reverse hidden layer state vector; F Lf is the hidden layer equation, and the hyperbolic tangent function is used as the activation function: Among them, x is the input, that is, and F Lo is the output layer equation, using the ReLU function as the activation function: Among them, x is the input, that is, W5h t +W6h t ', α = 0.03; Step 4.2: Calculate the mean square error loss L3 of the long short-term memory network according to the following formula: Step 4.3: Use two fully connected layers based on the attention mechanism to generate a one-dimensional scalar for each image feature vector, and then use the softmax function F S Generate aggregate attention weights, and calculate the image sorting feature vector based on the aggregate attention weights: Among them, W7 and W8 are the weights of the fully connected layer, b7 and b8 are the corresponding biases, and γ is the aggregate attention weight.

9. The dual-channel person re-identification method based on the attention twin network as claimed in claim 1, characterized in that: The step 5 is specifically as follows: Step 5.1: Add the loss L2 output by the second twin network to each image sorting feature vector element by element to obtain the corresponding fused feature vector; Step 5.2: Use L2 norm to normalize multiple fused feature vectors; Step 5.3: Calculate the distance between the normalized fusion features by using the cosine similarity method for the normalized fusion feature vector to obtain the fusion feature similarity matrix; Step 5.4: Perform weighted averaging on the fusion feature similarity matrix to obtain the aggregated similarity of each fusion feature; Step 5.5: Sort the images corresponding to the fusion features according to the similarity after aggregation, and output the sorted image list.

10. The dual-channel person re-identification method based on the attention twin network as claimed in claim 1, characterized in that: The overall loss L of the first twin network, the second twin network and the long short-term memory network is calculated by the following formula A : <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> A <h2 style=";text-align:left;direction:ltr"> (L1+L2+L3) By calculating L A Update the weights of the first twin network, the second twin network and the long short-term memory network to minimize the loss function, optimize the model parameters and achieve the purpose of pedestrian re-identification.