An occluded pedestrian re-identification method based on convolution and self-attention

Abstract

The application provides a kind of based on convolution and self-attention occluded pedestrian re-identification method, belong to computer vision field.The application is by combining the advantages of two different network structures of convolution operation and attention mechanism respectively, constructs double-branch encoder to carry out feature coding to pedestrian image, so that the model obtains rich semantic global feature and local feature with local details;And proposed a local information aggregation module, can further effectively suppress the noise in the feature map obtained by feature coding, to extract discriminative features;In addition, a multi-head global-to-local attention module is also designed to incorporate global information into local features, enriching the information contained in local features and enhancing the network's ability to extract discriminative local features, effectively improving the accuracy of the entire model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a method for re-identifying occluded pedestrians based on convolution and self-attention. Background Technology

[0002] Pedestrian re-identification aims to retrieve the same pedestrian captured by different cameras, and is a crucial visual task in computer vision with broad application prospects in smart cities. However, in real-world scenarios, pedestrian images are frequently affected by various factors such as viewpoint changes, resolution differences, and occlusion. Occlusion not only introduces interference information from other objects but also results in the loss of partial information about the target pedestrian image, severely impacting the performance of pedestrian re-identification systems. Due to incomplete information and spatial misalignment issues, occluded pedestrian re-identification is a widely discussed yet challenging research topic.

[0003] Most existing methods for pedestrian re-identification with occlusion are based on convolutional neural networks (CNNs). Because the receptive field of convolution operations is limited to a small Gaussian distribution region, CNNs are only good at extracting local features and have limited ability to capture global information, thus failing to generate effective pedestrian feature representations. In recent years, thanks to the self-attention mechanism, visual self-attention models (ViT) have been successfully applied to many computer vision tasks such as image classification, semantic segmentation, pose estimation, and pedestrian re-identification. ViT-based methods have strong global and dynamic receptive field modeling capabilities, effectively utilizing long-distance dependent semantic information and exhibiting good performance even under severe occlusion noise. However, when converting images into patch sequences, this method often lacks edge information, leading to insufficient local feature extraction capabilities. When pedestrians are severely occluded or similar to other backgrounds, misjudgments are prone to occur.

[0004] For occluded pedestrian re-identification, extracting pedestrian features requires not only rich and discriminative global semantic information, but also key information embedded in local features. Previous methods have failed to address both of these points, making it difficult to generate efficient pedestrian feature representations. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention proposes an occluded pedestrian re-identification method based on convolution and self-attention. This method effectively combines the advantages of two different network structures, CNN and ViT, and can simultaneously capture global features with long-distance dependence and local features with local details, thereby improving the model's ability to extract pedestrian information.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] An occluded person re-identification method based on convolution and self-attention includes the following steps:

[0008] Step 1: Use a dual-branch encoder to obtain global feature codes and local feature maps of pedestrian images;

[0009] Step 2: Input the local feature map of the pedestrian image into the local information aggregation module to obtain the local feature sequence;

[0010] Step 3: Divide the local feature sequence into N groups on average, and input the N groups of local feature sequences and the global feature encoding obtained in Step 1 into the multi-head global-to-local attention module layer and the forward feedback layer, respectively, to obtain N local feature encodings;

[0011] Step 4: Train the model using triplet loss and classification loss on the global feature encoding obtained in Step 1 and the N local feature encodings obtained in Step 3 to obtain the occluded pedestrian re-identification model;

[0012] Step 5: Load the model from Step 4, input the target pedestrian image to be queried into the trained occluded pedestrian re-identification model, and obtain the query result.

[0013] Furthermore, step 1 specifically includes:

[0014] Step 1.1: For the input pedestrian image, first use an encoder built by a ViT network based on a self-attention mechanism to encode it.

[0015] In constructing the input for the ViT network, the pedestrian image is first divided into N fixed-length and width P patch sequences using a sliding window method with a step size S. Then, the N patch sequences are mapped to the patch embedding vector space through linear projection. Next, a learnable classification token is connected to the patch embedding, and positional and camera encodings are added. The final input can then be represented by formula (1):

[0016] x in =Concat(x) class ,x patch )+Pos embedding +Cam embedding (1)

[0017] Where, x class Represents a category token, x patch Pos represents the patch sequence. embedding Indicates location encoding, Cam embedding This indicates the camera encoding, and Concat(*) indicates the concatenation operation.

[0018] Finally, x inThe input is fed into a ViT network consisting of multiple multi-head self-attention layers and a feedforward feedback layer for feature encoding, resulting in the global feature encoding of the pedestrian image, as shown in Equation (2):

[0019] x vit =ViT(x in (2)

[0020] Step 1.2: For the input pedestrian image, use an encoder built from a convolutional neural network based on convolution operations to encode it.

[0021] The convolutional neural network used is ResNet50, a common type in pedestrian re-identification, with the average pooling layer, fully connected layer, and final downsampling operation removed. For the input pedestrian image, the encoder encodes it to obtain the local feature map of the pedestrian image, as shown in formula (3):

[0022] x cnn =Resnet50(x) (3)

[0023] Where x represents the input pedestrian image.

[0024] Step 1.3: Combine the encoders described in Steps 1.1 and 1.2 in parallel to obtain a dual-branch encoder, and encode the output global feature x. vit and local feature map x cnn Together they serve as feature encodings for pedestrian images.

[0025] Furthermore, step 2 specifically includes:

[0026] Step 2.1: For the local feature map x obtained in Step 1 cnn A 1×1 convolutional layer is used, and its output is converted into token form. The output of this step is represented by formula (4):

[0027] y1 = Reshape(Conv) 1×1 (x cnn (4)

[0028] Where Reshape represents the feature form transformation operation, Conv 1×1 This represents a convolutional layer of size 1×1, x cnn This is the local feature map obtained in step 1.

[0029] Step 2.2: After the convolutional block described in Step 2.1, add a fully connected layer and transform its output format. This operation can increase cross-channel information interaction, increase nonlinearity, and improve the network's expressive power. The output of this step is expressed by formula (5):

[0030] y2 = Reshape(Linear(y1)) (5)

[0031] Here, Reshape represents the feature form transformation operation, and Linear represents a fully connected layer.

[0032] Step 2.3: Add a 3×3 depthwise convolutional layer to achieve information exchange between different tokens, and similarly, change its output format. Depthwise convolution can be performed independently on each channel of the input layer, effectively utilizing feature information in the same space. The output of this step is expressed by formula (6):

[0033] y3 = Reshape(DWConv) 3×3 (y2)) (6)

[0034] Where Reshape represents the feature form transformation operation, DWConv 3×3 This represents a depthwise convolutional layer of size 3×3.

[0035] Step 2.4: After the deep convolutional layer, a fully connected layer is applied, and the feature representation is obtained in the form of tokens. The output of this step is expressed by formula (7):

[0036] y liam =Linear(y3) (7)

[0037] Where Linear represents a fully connected layer, y3 is the output of step 2.3, and y liam This is the final output of the local information aggregation module.

[0038] Furthermore, step 3 specifically includes:

[0039] Step 3.1: For the feature sequence y obtained from the local information aggregation module in step 2... liam Divide it into N equal parts, which are the N local features Z of the pedestrian image. i , where i = 1, 2, ..., N.

[0040] Step 3.2: For N local features, to avoid information distortion caused by camera bias, camera encoding is added to each feature to obtain the local feature sequence. Where i = 1, 2, ..., N, this step is expressed by formula (8):

[0041]

[0042] In the formula, Cam embedding This indicates the camera encoding.

[0043] Step 3.3: For each local feature, learn the local feature and the global feature x obtained in Step 1. vit Simultaneously, the data is fed into the weight-sharing multi-head global-to-local attention module and the forward feedback layer to learn local features containing global information.

[0044] The multi-head global-to-local attention module is constructed from 16 attention modules, and its structure can be represented by the following formula:

[0045] z(x1,x2)=Concat(f(x1),g(x2)) (9)

[0046]

[0047] head i =GLA(zW i Q ,zW i K ,zW i V (11)

[0048] MHGLA(x1,x2)=Concat(head1,…,head 16 W H (12)

[0049] In the formula, x1 and x2 represent two different input features, Q, K, and V represent the query set, key set, and value set, respectively, d represents the scaling factor, and W... i * and W H All are trainable matrices. Concat(*) represents the concatenation operation, f(*) and g(*) represent selecting the patch sequence and classification token from the feature sequence, respectively, and MHGLA(*) represents the multi-head global to local attention module.

[0050] Overall, taking local and global features as input, the multi-head global-to-local attention module outputs the encoded features. and the encoded features output by the forward feedback layer It can be expressed by the following formula:

[0051]

[0052]

[0053] In the formula, LN represents layer normalization, MHGLA represents multi-head global to local attention module, and FFN represents forward feedback layer.

[0054] Furthermore, step 4 specifically includes:

[0055] Step 4.1: During the training phase, use the hard triplet loss L tri and cross-entropy loss L cls To train the model. For global features containing long-range dependency information, its learning objective is expressed by formula (15):

[0056] L global =L cls (f g )+L tri (f g (15)

[0057] Among them, f g L represents global features. global The loss function represents the global features.

[0058] Step 4.2: For the local features of N partial pedestrian images containing local details, the hard triplet loss and cross-entropy loss are also used for training, expressed by formula (16):

[0059]

[0060] Among them, f l i L represents the local features of each part of the pedestrian image. local The loss function represents the local features.

[0061] Step 4.3: Finally, the model is trained by combining global and local features simultaneously. This allows the model to learn feature representations with rich information, thereby maximizing the accuracy of the entire model. The total loss function is shown in the following equation:

[0062] L = L global +L local (17)

[0063] Furthermore, step 5 specifically includes:

[0064] Load the model trained in step 4, input the pedestrian image to be queried into the model, and output the retrieval results and corresponding evaluation metrics.

[0065] The beneficial effects of this invention are as follows: This invention constructs a dual-branch encoder to encode the features of pedestrian images, thereby enabling the model to obtain global features with rich global semantics and local features with local details; it also proposes a local information aggregation module, which can further suppress noise in the feature map and extract discriminative features; in addition, a multi-head global-to-local attention module is designed to integrate global information into local features, thereby enhancing the network's ability to extract discriminative local features. Attached Figure Description

[0066] Figure 1 This is a diagram illustrating the overall framework of an occluded pedestrian re-identification network based on convolution and self-attention.

[0067] Figure 2 This is a local information aggregation module.

[0068] Figure 3 This is a multi-head global-to-local attention module. Detailed Implementation

[0069] The embodiments of the present invention are implemented under the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given. However, the protection scope of the present invention is not limited to the following embodiments.

[0070] This embodiment uses Linux as the development environment, PyCharm as the development platform, and Python as the development language. It employs the occluded pedestrian re-identification method based on convolution and self-attention of the present invention to complete the retrieval of specific pedestrians in occluded scenes.

[0071] In this embodiment, the occluded pedestrian re-identification method based on convolution and self-attention includes the following steps:

[0072] Step 1: Load the pre-trained weights of the dual-branch encoder network into... Figure 1 In the occluded pedestrian re-identification network shown;

[0073] Step 2: Input the training set into the occluded pedestrian re-identification network from Step 1 for training;

[0074] Step 3: Using the target pedestrian image as input, load the model trained and saved in Step 2 to obtain the corresponding search results and evaluation metrics. This invention uses Rank-1 and mAP as evaluation metrics, where Rank-1 represents the probability that the first image in the search results contains the correct result, and mAP represents the mean accuracy.

[0075] Based on the above steps, this invention is compared with Part-Alighed models, PCB models, PVPM models, PGFA models, HOReID models, ISP models, PAT models, QPM models, TransReID models, PED models, PFT models, and PFD models. As shown in Table 1, the method proposed in this invention outperforms other methods in both Rank-1 and mAP metrics on common test sets.

[0076] Table 1 compares the model's performance on the Octcluded-Duke dataset.

[0077]

[0078] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for re-identifying occluded pedestrians based on convolution and self-attention, characterized in that, The method includes the following steps: Step 1: Use a dual-branch encoder to obtain global feature codes and local feature maps of pedestrian images; Step 2: Input the local feature map of the pedestrian image into the local information aggregation module to obtain the local feature sequence; Step 2.1: For the local feature map obtained in Step 1 Use a size of The convolutional layer is converted into token form, and the output of this step is represented by formula (4): ； in, This indicates a characteristic form conversion operation. Indicates size is convolutional layers, This is the local feature map obtained in step 1; Step 2.2: After the convolutional layer described in Step 2.1, add a fully connected layer and transform its output form; the output of this step is expressed by formula (5): ； in, This indicates a characteristic form conversion operation. Indicates a fully connected layer; Step 2.3: Add a size of The deep convolutional layer enables information exchange between different tokens, and similarly, changes its output format; the output of this step is expressed by formula (6): ; in, This indicates a characteristic form conversion operation. Indicates size is Deep convolutional layers; Step 2.4: After the deep convolutional layer, a fully connected layer is used to obtain the feature representation in the form of tokens; the output of this step is expressed by formula (7): ； in, Indicates a fully connected layer. The output of step 2.3, This is the final output of the local information aggregation module; Step 3: Divide the local feature sequence into N groups on average, and input the N groups of local feature sequences and the global feature encoding obtained in Step 1 into the multi-head global-to-local attention module layer and the forward feedback layer, respectively, to obtain N local feature encodings; Step 3.1: For the feature sequence obtained from the local information aggregation module in Step 2... Divide it into equal parts Each part serves as a pedestrian image. Local features ,in ; Step 3.2: For To avoid information distortion caused by camera bias, camera encoding is added to each local feature, resulting in a local feature sequence. ,in This step is expressed by formula (8): ； In the formula, Indicates camera encoding; Step 3.3: For each local feature, the local features are compared with the global features obtained in Step 1. Simultaneously, the data is fed into the weight-sharing multi-head global-to-local attention module and the forward feedback layer to learn local features containing global information; The multi-head global-to-local attention module is constructed from 16 attention modules, and its structure is represented by the following formula: ； ； ； ； In the formula, This represents two different input features. , , These represent the query set, key set, and value set, respectively. Indicates the scaling factor. and They are all trainable matrices. This indicates a splicing operation. and These represent selecting the patch sequence and the classification token from the feature sequence, respectively. This represents a multi-head global-to-local attention module; Overall, taking local and global features as input, the multi-head global-to-local attention module outputs the encoded features. and the encoded features output by the forward feedback layer Expressed using the following formula: ； In the formula, Representation layer normalization, This represents a multi-head global-to-local attention module. Indicates a forward feedback layer; Step 4: Train the model using triplet loss and classification loss on the global feature encoding obtained in Step 1 and the N local feature encodings obtained in Step 3 to obtain the occluded pedestrian re-identification model; Step 5: Load the model from Step 4, input the target pedestrian image to be queried into the trained occluded pedestrian re-identification model, and obtain the query result.

2. The occluded person re-identification method based on convolution and self-attention according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1: For the input pedestrian image, first use an encoder built by a ViT network based on a self-attention mechanism to encode it; In constructing the input to the ViT network, the pedestrian image is first divided into N fixed-length and width P patch sequences with a step size S using a sliding window approach. Then, the N patch sequences are mapped to the patch embedding vector space through linear projection. Next, a learnable classification token is connected to the patch embedding, and position encoding and camera encoding are added. The final input is then represented by formula (1): ； in, Indicates a category token. Indicates a patch sequence. Indicates position code, Indicates camera code, Indicates a splicing operation; Finally The input is fed into a ViT network consisting of multiple multi-head self-attention layers and a feedforward layer for feature encoding, resulting in the global feature encoding of the pedestrian image, as shown in formula (2): ； Step 1.2: For the input pedestrian image, use an encoder built from a convolutional neural network based on convolution operations to encode it; Among them, the type of convolutional neural network selected is ResNet50, which is commonly used in pedestrian re-identification. The average pooling layer, the fully connected layer, and the final downsampling operation are removed. For the input pedestrian image, after encoding by the encoder, the local feature map of the pedestrian image is obtained, as shown in formula (3): ; in, This represents the input pedestrian image; Step 1.3: Combine the encoders described in Steps 1.1 and 1.2 in parallel to obtain a dual-branch encoder, and encode the output global features. and local feature maps Together they serve as feature encodings for pedestrian images.

3. The occluded pedestrian re-identification method based on convolution and self-attention as described in claim 1 or 2, characterized in that, Step 4 specifically includes: Step 4.1: During the training phase, use hard triplet loss. and cross-entropy loss To train the model; for global features containing long-range dependency information, its learning objective is expressed by formula (15): ； in, Represents global features. The loss function representing global features; Step 4.2: For the local features of N partial pedestrian images containing local details, the hard triplet loss and cross-entropy loss are also used for training, expressed by formula (16): ； in, Represents the local features of each part of the pedestrian image. Loss function representing local features; Step 4.3: Finally, the model is trained by combining global and local features simultaneously. This allows the model to learn feature representations with rich information, thereby maximizing the accuracy of the entire model. The total loss function is shown in the following formula: 。

Images