Method for occluded pedestrian re-identification based on convolutional neural network optimized ViT

Abstract

The application discloses a method for combining a convolutional neural network to optimize a ViT for occluded pedestrian re-identification, which combines a convolutional neural network to optimize the network structure of a Vision Transformer, and combines the advantages of CNN and Transformer. The application designs a network structure with double branches of CNN and Transformer, so that the features extracted from pictures have the advantages of both Transform and CNN. In this way, the global relationship between tokens is obtained through Transform, and the local information of tokens is obtained through CNN. Finally, the local information and global information are fused to obtain robust features. Finally, the model is iteratively optimized by combining the triplet loss and cross-entropy loss, so that the final loss of the model is reduced to a minimum value, and the accuracy of the features and the performance of the model are improved. In this way, the problem of pedestrian re-identification under the occlusion condition is effectively solved, and a more efficient framework is provided for the actual application of occluded pedestrian re-identification.

Description

[0001] This invention relates to the field of computer vision technology, and in particular to a method for optimizing a convolutional neural network for pedestrian re-identification, specifically a Vision Transformer (ViT). Background Technology

[0002] Pedestrian re-identification is a highly challenging task, falling under the category of image processing and analysis in complex video environments. The goal of pedestrian re-identification is to search for a target pedestrian among multiple non-overlapping camera feeds. Due to the increasing number of surveillance cameras and the urgent need for public safety, pedestrian re-identification can play a significant role in intelligent surveillance systems, autonomous driving, intelligent security, and criminal investigation, possessing substantial research and practical significance.

[0003] In recent years, the widespread success of convolutional neural networks (CNNs) in the field of computer vision has become commonplace. CNNs are widely used to address many specific challenges, such as situations where pedestrians are occluded by other pedestrians or objects, or where pedestrians are misaligned, body parts are incomplete, or there is background interference or changes in viewpoint. Due to these challenges, simply using the CNN backbone for feature extraction currently does not significantly improve pedestrian re-identification performance.

[0004] Transformers are deep neural networks based on self-attention mechanisms, initially applied in natural language processing. Due to their powerful performance, researchers have transferred them to computer vision, with ViT (Vision Transformer) being the most famous example, achieving performance comparable to traditional convolutional neural networks. While ViT models like Transreid are now used for person re-identification, their drawbacks include the need for large amounts of pedestrian data and significant GPU performance during training. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for pedestrian re-identification under occlusion that combines convolutional neural networks with ViT optimization. Its purpose is to solve the problem of noisy or misaligned pedestrian features due to occlusion, namely, by extracting both local and global features, and combining the advantages of convolutional neural networks and ViT to extract more robust and discriminative features.

[0006] To achieve the above objectives, the present invention provides a method for occluded pedestrian re-identification that combines convolutional neural network optimization of ViT, comprising the following steps;

[0007] S1. Construct the network framework, which consists of two parts: CNN and ViT;

[0008] S2. While using the CNN network to extract local features, the ViT network is also used to extract global feature information.

[0009] S3. Fuse the local features after linear projection and the global features after stretching and transposing.

[0010] S4. Use the optimized transformer architecture to strengthen the connections between features, and feed the features obtained in S3 into the transformer's self-attention module.

[0011] S5. Finally, calculate the cross-entropy loss and triplet loss based on the output features, and continuously optimize the model based on the losses.

[0012] This invention provides a method for occluded pedestrian re-identification that combines convolutional neural networks with optimized ViT. Compared with existing technologies, it has the following advantages:

[0013] This invention addresses the problems of noise in pedestrian features and misalignment caused by occlusion. It optimizes the existing Vision Transformer model using a convolutional neural network. Because the Vision Transformer has a self-attention module, it allows the model to focus more on globally relevant features and ignore occluded parts. Therefore, combining the advantages of both convolutional neural networks and the Vision Transformer effectively solves the problem of noise in pedestrian features caused by occlusion, improving the robustness and accuracy of re-identifying occluded pedestrian features. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is the overall network structure diagram of the method for re-identifying occluded pedestrians using a convolutional neural network optimized ViT in this invention patent.

[0016] Figure 2 This is a structural diagram of the convolutional neural network optimization module in this invention patent. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0018] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. A Vision Transformer for pedestrian re-identification under occlusion includes steps S1 to S5:

[0019] S1. Construct the network framework, which consists of two parts: CNN and ViT;

[0020] S2. While using the CNN network to extract local features, the ViT network is also used to extract global feature information.

[0021] S3. Fuse the local features after linear projection and the global features after stretching and transposing.

[0022] S4. Use the optimized transformer architecture to strengthen the connections between features, and feed the features obtained in S3 into the transformer's self-attention module.

[0023] S5. Finally, calculate the cross-entropy loss and ternary combination loss based on the output features, and continuously optimize the model based on the losses.

[0024] The following is a detailed description of each step.

[0025] In step S1, the network architecture is constructed, such as... Figure 1 As shown, this network consists of two parts: CNN and ViT. The specific steps are as follows:

[0026] S101. Construct a framework that combines CNN and ViT, consisting of two branches;

[0027] S102. Feed the pedestrian image into the CNN branch, and simultaneously flatten it and feed it into the ViT branch to extract features.

[0028] In step S2, while using a CNN network to extract local features, a ViT network is also used to extract global feature information. The specific steps are as follows:

[0029] S201. The feature extraction part of the ViT branch adopts the traditional ViT structure, as shown below:

[0030] Y = [y 1 ,y 2 ,···,y i,···,y n ],yi=Conv2d(X i ), 1≤i≤n (1)

[0031] Among them, X i This represents the i-th part after dividing the image into n parts, y i Let Y represent the word embedding of the i-th part, Conv2d represent a convolutional layer, and Y represent X. i The y formed after the convolutional layer i A set;

[0032] Y′=transpose(flatten(Y)) (2)

[0033] Then, Y is flattened and transposed to obtain Y′, where transpose represents matrix transpose operation, flatten represents matrix flattening operation, and Y′ represents the result after flattening and transposing operations;

[0034]

[0035] Among them, C T A token used to represent a category. T represents the matrix addition operation. trans This represents the final feature obtained through this branch, namely Y′ and C. T The sum of;

[0036] S202. Simultaneously, input the pedestrian image into the CNN to extract local features;

[0037] X′=CNN tokensblock (X), X∈R B,C,H,W ,X′∈R B,C′,H′,W′ (4)

[0038] Where X represents the pedestrian image, R represents the image dimension, B refers to the batch size of the pedestrian images (i.e., the number of input images per batch), C represents the number of image channels, H represents the image height, and W represents the image width. CNN tokensblock This represents the optimization module of the convolutional neural network, and X′ represents the value after passing through the CNN. tokensblock The processed image, where B, C′, H′ and W′ represent the batch, number of channels, image height and image width after processing, respectively;

[0039]

[0040] Among them, T cnn C represents the local features extracted after the convolutional network optimization module. TSimilar to the previous branch, it is also a token used for classification. conv1 and conv2 are convolutional neural network layers, and FC represents a fully connected layer.

[0041] In step S3, it is necessary to fuse the local features after linear projection and the global features after stretching and transposing.

[0042] S301. First, the features obtained from the two branches in step S2 are fused by directly adding the two features together.

[0043] T = T CNN +T trans (6)

[0044] Where T represents the final fusion feature that combines the features of the CNN branch and the ViT branch.

[0045] In step S4, the optimized transformer architecture is used to strengthen the connections between features. The features obtained in S3 are then fed into the transformer's self-attention module.

[0046] S401. Add the location embedding PE to the fused features obtained in step S3.

[0047]

[0048] in This indicates that the features have been added after location embedding.

[0049] S402. Use the self-attention module of the transformer to enhance the connection between features and capture global information. The output of the module is obtained by performing pairwise operations on the inputs to obtain weights and then performing a weighted sum on the inputs.

[0050]

[0051] Where Q, K, V are the input matrices The results obtained by performing different linear transformations, Attention represents the self-attention module, softmax represents the activation function, and K T This represents the result of transposing the K matrix, where d represents the dimensions of Q and K.

[0052] S403, Add the result obtained in S402. It itself, then goes through another MLP layer to obtain Then, in addition to what it itself received The final result is obtained after passing through multiple transformer layers.

[0053]

[0054] Where L represents the number of L transformer layers. This represents the result after passing through layer i, LN represents LayerNomal layer, MLP represents multilayer perceptron, and MSA represents multi-head self-attention module.

[0055] In step S5, the cross-entropy loss and triplet loss are calculated based on the output features, and the model is continuously optimized based on the losses.

[0056] S501. The loss function of this network structure model includes cross-entropy loss and triplet loss. Each pedestrian is treated as a category, and the cross-entropy function L used for classification... CE The definition is as follows:

[0057]

[0058] Where, y′ i Let y represent the probability of being predicted as class i. i Let C represent the true value, and C represent the total class. Cross-entropy loss makes the predicted value infinitely close to the true value. In addition, triplet loss is used to reduce intra-class distance and increase inter-class distance, and its definition is as follows:

[0059]

[0060] Among them, L tri The meaning is triplet loss. This represents the distance between negative sample pairs, i.e., the distance between the target image and the least similar image. This represents the distance between positive sample pairs, i.e., the distance between the target image and the most similar image, where m represents the necessary parameter margin and N represents the number of samples;

[0061] The final loss is the sum of the triplet loss and the cross-entropy loss:

[0062] Loss = L CE +L tri (12)

[0063] S502. The training process of the network model is an iterative process of continuously optimizing by reducing the loss. That is, after training for one round, the loss obtained in this round is fed back into the model, and the next round of training is carried out. After many such iterations, the loss is reduced, thereby obtaining more robust features.

[0064] This invention combines a convolutional neural network with an optimized Vision Transformer network structure, offering the following advantages: It integrates the strengths of both CNN and ViT. Since CNN possesses translation invariance, while Transformer focuses more on the overall picture, this invention designs a dual-branch network structure of CNN and ViT. This allows the extracted image features to possess the advantages of both Transformer and CNN. This approach captures the global relationships between tokens through Transformer and the local information of tokens through CNN, ultimately fusing the local and global information to obtain robust features. Simultaneously, this effectively solves the problem of pedestrian re-identification under occlusion conditions, constructing a new and effective method for occluded pedestrian re-identification, providing a more efficient framework for practical applications of occluded pedestrian re-identification.

[0065] This invention also proposes a method for occluded pedestrian re-identification that combines convolutional neural networks to optimize ViT, comprising:

[0066] A network framework is constructed, which consists of two parts: CNN and ViT.

[0067] The given pedestrian image is input into a CNN network to extract local features, and the pedestrian image is flattened into a sequence and then input into a ViT network to extract global feature information.

[0068] The local features after linear projection and the global features after stretching and transposing are fused together.

[0069] The optimized transformer architecture is used to strengthen the connections between features, and the features obtained in the previous step are fed into the transformer's self-attention module.

[0070] Finally, cross-entropy loss and ternary combination loss are calculated based on the output features, and the model is continuously optimized based on the losses.

[0071] The present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0072] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for occluded pedestrian re-identification combining convolutional neural network optimization ViT, characterized in that, The method is executed by a computer and includes the following steps: S1, constructing a network framework composed of CNN and ViT, the input data of the network framework being a pedestrian image, and the specific implementation process of S1 being as follows: S101, constructing a framework in which CNN and ViT coexist, composed of two branches; S102, sending the pedestrian image into the CNN branch and simultaneously flattening and sending into the ViT branch to extract features; S2, extracting local feature information using the CNN network and global feature information using the ViT network at the same time; S3, fusing the local feature after linear projection and the global feature after stretching and transposition operation; S4, using an optimized transformer architecture to strengthen the connection between features, sending the features obtained in S3 into the self-attention module of the transformer, and the specific implementation process of S4 being as follows: S401, adding position embedding PE to the fused features obtained in step S3, wherein denotes the features after having added the position embedding, S402, using the self-attention module of the transformer to enhance the connection between features and capture global information, and the output result of the module being obtained by weighting summation after weighting the input result two by two to obtain the weight; where Q, K, V are matrices of input The results of different linear transformations, Attention represents the self-attention module, and softmax represents the activation function T represents the result of transposing the K matrix, and d represents the dimension of Q and K S403, add the result obtained in S402 to itself After that, add itself to obtain After that, add itself to obtain After multiple layers of transformer layers, obtain the final result wherein L represents L layers of transformer layers, denotes the result after i layers, LN denotes a LayerNomal layer, MLP denotes a multi-layer perceptron, and MSA denotes a multi-head self-attention module. S5, finally calculating the cross-entropy loss and the triple combination loss according to the output features, and continuously optimizing the model according to the loss.

2. The method for pedestrian re-identification occlusion with convolutional neural network optimized ViT according to claim 1, wherein, The specific implementation process of S2 is as follows: S201, the feature extraction part of the ViT branch adopts the traditional ViT structure, and the expression is as follows: Y = [y 1 ,y 2 ,···,y i ,···,y n ], yi = Conv2d(X i ), 1≤i≤n (1) wherein X i represents the i-th part of the image divided into n parts, y i represents the word embedding of the i-th part, Conv2d represents a convolutional layer, Y represents the set of y i formed after passing through the convolutional layer i ; Y' = transpose (flatten (Y)) (2) Then Y is first flattened and then transposed to obtain Y', wherein transpose represents the matrix transposition operation, flatten represents the matrix flattening operation, and Y' represents the result after the flattening and transposition operation; T trans = Y' 0 C T (3) wherein C T represents a token for classification, and represents a matrix addition operation, and T trans represents a feature obtained through this branch, i.e., the sum of Y′ and C T . S202, the pedestrian image is simultaneously input into the CNN to extract local features; X' = CNN tokensblock (X), X e R B,C,H,W , X' e R B,C′,H′,W′ (4) wherein X represents the pedestrian image, R represents the dimension of the image, B represents the batch of the pedestrian image, i.e. the number of input images each time, C represents the channel number of the image, H represents the height of the image, and W represents the width of the image, CNN tokensblock represents a convolutional neural network optimization module, X' represents the image processed by the CNN tokensblock , B, C', H' and W' represent the batch, the channel number, the height of the image and the width of the image of the processed image, respectively. T CNN = FC(conv2(conv1(X′))) ⊕ C T (5) wherein T CNN represents the local feature extracted after the convolutional network optimization module, C T The same as the last branch, it is also a token for classification, and conv1 and conv2 are convolutional neural network layers, and FC represents a fully connected layer.

3. The method for pedestrian re-identification using a combined convolutional neural network optimized ViT according to claim 1, wherein, The specific implementation process of S3 is as follows: S301, first, the features obtained from the two branches in step S2 are fused, and the form is to directly add the two features, T = T CNN + T trans (6) Wherein, T represents the fused features combined with the CNN branch features and the ViT branch features.

4. The method for pedestrian re-identification using a combined convolutional neural network optimized ViT according to claim 1, wherein, The specific implementation process of S5 is as follows: S501, the loss function of the network structure model includes cross-entropy loss and triplet loss, each pedestrian is regarded as a category, and a cross-entropy function L for classification CE is defined as follows: Where, y′ i Let y represent the probability of being predicted as class i. i Let C represent the true value, and C represent the total class. Cross-entropy loss makes the predicted value infinitely close to the true value. In addition, triplet loss is used to reduce intra-class distance and increase inter-class distance, and its definition is as follows: wherein L tri The meaning of L represents the distance of negative sample pairs, i.e. the distance of the target image and the least similar image, represents the distance of positive sample pairs, i.e. the distance of the target image and the most similar image, m represents a parameter margin, and N represents the number of samples; The final loss Loss is the sum of the triple combination loss and the cross-entropy loss: Loss = L CE + L tri (12) S502, the training process of the network model is a process of continuously optimizing iteration by reducing the loss, that is, after one round of training, the loss obtained in this round is fed back to the model for the next round of training, and after multiple such repeated iterations, the loss is reduced to obtain more robust features.

Images