Pedestrian re-identification method with feature feedback and electronic device

Abstract

The application discloses a pedestrian re-identification method with feature back transmission and electronic equipment, wherein the method comprises the following steps: acquiring an input image, performing feature sharpening processing on the input image; inputting the processed image into a trained pedestrian re-identification model to output a pedestrian re-identification result; the pedestrian re-identification model comprises a transformer encoder and a feature back transmission module; the encoder is used for two-round training to extract global information of a pedestrian image; the feature back transmission module is used for enhancing shallow features of the first-round training, aggregating global and local features and emphatically enhancing the local features, and performing second-round training on the feature back transmission network to supplement and strengthen missing detailed information. The application highlights pedestrian features by sharpening image data, improves the utilization of shallow features, allows the model to focus on specific areas based on previous results to improve accuracy, effectively improves the performance of the pedestrian re-identification model, and can be widely applied to the technical field of image recognition.

Description

Technical Field

[0001] This invention relates to the field of computer vision image recognition technology, and in particular to a pedestrian re-identification method and electronic device based on feature feedback. Background Technology

[0002] Person re-identification (Re-ID) is an important research area in computer vision, aiming to identify and match the same pedestrian under different cameras. With the popularization of intelligent surveillance systems and the surge in data volume, person re-identification is becoming increasingly important in fields such as security monitoring, public safety, intelligent transportation, business analytics, and human-computer interaction.

[0003] Pedestrian re-identification is a challenging task. Variations in lighting, viewing angle, cluttered backgrounds, and the diversity of pedestrian appearances (such as clothing, posture, and body type) can cause the same pedestrian to appear significantly different on different cameras. Effectively addressing these issues is crucial for improving the intelligence level of the system. To solve these problems, researchers have begun exploring various technologies and algorithms to improve the accuracy and robustness of re-identification.

[0004] In recent years, significant progress has been made in pedestrian re-identification research. Currently, deep learning-based methods dominate the field, requiring the training of a deep learning model to extract pedestrian features for classification. Training a deep learning model for pedestrian re-identification can be divided into the following steps: First, a set of pedestrian images from the training set is processed and input into a deep neural network. After extensive computation, a set of feature vectors representing pedestrians is obtained. These vectors are then substituted into a pre-designed loss function to calculate the loss value. Finally, the model parameters are updated through gradient backpropagation and an optimizer. Depending on their focus, researchers may choose to study four aspects: dataset, representation learning, metric learning, and optimization ranking. Among these, metric learning, as the feature extraction part, has been the most studied, and designing an effective network model for extracting pedestrian features is crucial.

[0005] Pedestrian re-identification is mainly studied from the aspects of global features, local features, auxiliary features and attention features: (1) Pedestrian re-identification method based on global image is the main research method in the early stage and is relatively simple. The image is directly input into the CNN network for convolution to obtain global appearance features such as clothing, body shape and posture of pedestrians and then classify them. This approach achieved good results at the beginning, but in the inevitable downsampling process, the resolution will gradually decrease and become irreversible, making the network easy to ignore local discriminative features, and directly extracting features from the whole image is easily affected by occlusion, background and human posture, thus affecting the recognition rate in these situations. (2) Pedestrian re-identification method based on local features divides the image into regions or estimates the posture to extract feature information, prompting the network to pay more attention to local regions. The network can use local information for matching and classification, achieving good results, but under severe occlusion, even local information contains a lot of interference noise, and it is powerless to deal with misaligned people. (3) Pedestrian re-identification methods based on auxiliary features utilize auxiliary information such as semantic information, camera information, GAN, and information enhancement to enhance feature information. This strategy significantly improves the accuracy of recognition, but due to the introduction of additional information, it inevitably leads to a large consumption of computing power, and the final recognition rate depends to a large extent on the cleanliness of the pedestrian information extracted in the first stage. There is a possibility that discriminative fine-grained features on pedestrians are treated as interference and classified as background information, resulting in the loss of key pedestrian information. (4) Pedestrian re-identification methods based on attention mechanisms have many attention mechanisms based on convolutional operations that can discover the most discriminative regions in pedestrian images. However, due to the limitation of the receptive field of the convolution kernel and the resolution loss caused by downsampling, it can usually only focus on a small region and cannot solve the problem of long-term dependence. Summary of the Invention

[0006] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a pedestrian re-identification method, electronic device and medium for feature backhaul.

[0007] The first technical solution adopted in this invention is:

[0008] A pedestrian re-identification method based on feature feedback includes the following steps:

[0009] Acquire the input image and perform feature sharpening processing on it;

[0010] The processed image is input into the trained pedestrian re-identification model, and the pedestrian re-identification result is output.

[0011] The pedestrian re-identification model includes a transformer encoder and a feature backpropagation module. The transformer encoder is used to perform two rounds of training to extract global information from pedestrian images. The feature backpropagation module is used to enhance the shallow features from the first round of training, aggregate global and local features and focus on enhancing local features, and then backpropagate the feature backpropagation network for the second round of training to supplement and enhance the missing detailed information.

[0012] Furthermore, a feature sharpening module is used to perform feature sharpening processing on the input image;

[0013] The feature sharpening module includes a deep convolutional layer, a sharpening unit, and a multilayer perceptron block.

[0014] The feature sharpening module works as follows:

[0015] For an input image I∈R H×W×C Where H represents height, W represents width, and C represents the number of channels, a depthwise convolutional layer with both length and width P is used to perform channel-wise convolution and point-wise convolution on the input image I to obtain image features I′; image features I′ are then input into a sharpening unit for feature sharpening, and finally, multilayer perceptron operation is performed, as shown in the following formula:

[0016] I′=DWConv(I)

[0017] I″ = MS(I′)

[0018] X = MLP(I″)

[0019] In the formula, DWConv(·) represents a deep convolutional layer, MS(·) represents a sharpening unit, and MLP(·) represents a multilayer perceptron operation.

[0020] Furthermore, the sharpening unit includes a grouped convolutional layer, a boundary enhancement module, and a regular convolutional layer;

[0021] The sharpening unit operates as follows:

[0022] First, two grouped convolutional layers with different dilation values ​​are used to extract the contextual features of image feature i′. Simultaneously, image feature I′ is passed to the boundary enhancement module to obtain boundary enhancement features. Finally, the contextual features and boundary enhancement features are concatenated along the channel dimension, and a 1×1 convolutional layer is used to obtain the enhanced feature representation, ensuring that the output image features remain the same size as the input, thus obtaining the pedestrian sharpening image features X∈R. H×W×C The formula is expressed as follows:

[0023]

[0024] I″2=BE(I′)

[0025]

[0026] In the formula, Group Conv(·) represents a grouped convolutional layer. Indicates addition, BE(·) represents a boundary enhancement module, and Conv(·) represents a regular convolutional layer. This indicates the concatenation of two feature vectors; I″1 represents the context feature, and I″2 represents the boundary enhancement feature.

[0027] Furthermore, the boundary enhancement module operates as follows:

[0028] First, a deep convolutional layer is used to encode the local representation of image feature I′. The structural information in each channel is enhanced to obtain feature Z, and the mean value is calculated to obtain the average response of the feature map. To highlight details, the average response is subtracted from image feature I′ to obtain feature Y. Furthermore, the spatial mean value of each channel of feature Z is calculated, and a softmax operation is used to obtain the sharpening factor s for each channel of feature map Y. The sharpening factor s is multiplied by feature Y and added to feature Z to obtain the sharpened boundary enhancement feature, as shown in the following formula:

[0029] Z = DW Conv(I′)

[0030] Y = I′ - C Mean(Z)

[0031] s = softmax(S Mean(Z))

[0032]

[0033] In the formula, C Mean(·) represents the mean calculated in the channel dimension, S Mean(·) represents the mean calculated in the spatial dimension, softmax represents the normalization operation, and ⊙ represents the addition.

[0034] Furthermore, the transformer encoder is used to perform the following processing on the pedestrian feature-sharpened image:

[0035] A convolutional layer containing C′ kernels of length and width P and stride S is used to process the input image features X∈R. H×W×C Perform a linear mapping to obtain N sequences of length D. i∈1,2,...,N, where:

[0036]

[0037] D = H * W * C

[0038] Generate a sequence x PLearnable cls tags of equal length, and the cls tags are associated with the sequence x. P The spatial information is combined and integrated into the combination through learnable positional encoding to obtain the input sequence z0, as shown in the following formula:

[0039]

[0040] In the formula, x cls ∈R 1×D Indicates the cls marker, E pos ∈R (N+1)×D Represented as positional encoding;

[0041] After the linear mapping is completed, the input sequence z0 is fed into four transformer modules. Each transformer module contains three transformer encoders. The encoder part includes a multi-head self-attention module and a multilayer perception module. The formula for calculating the output sequence is as follows:

[0042] z′ l =MSA(LN(z) l-1 ))+z l-1

[0043] z l =MLP(LN(z′) l ))+z′ l

[0044] In the formula, LN(·) represents the inter-layer normalization operation, MSA(·) represents the multi-head attention operation, and MLP(·) represents the multi-layer perception operation.

[0045] Furthermore, the feature backpropagation module consists of three convolutional branches, one pooling branch, a Squeeze-and-Excitation (SE) module, and a feedforward neural network (FFN);

[0046] The feature backpropagation module performs the following operations on the image features output from the first round of transformer training:

[0047] After concatenating the output feature maps s1 and s2 from the first and second transformer modules, multiplying them by the channel sharpening factor s, and then dividing them into four components, each processed through three convolutional branches and pooling branches, followed by Squeeze-and-Excitation modules and feedforward neural network operations, the result is concatenated with the pedestrian sharpened image feature X and fed back to the transformer encoder for the second round of model training. The formula is as follows:

[0048]

[0049] S1, S2, S3, S4 = Split(S)

[0050]

[0051] In the formula, Split(·) represents the channel separation operation, and FFN(SE(·)) represents passing through the Squeeze-and-Excitation module first, and then performing the feedforward neural network.

[0052] Furthermore, after obtaining the feature X′, the feature X′ is linearly mapped to obtain the sequence x. P The cls tag is associated with the sequence x. P The spatial information is combined and integrated into the combination through learnable positional encoding to obtain the input sequence z0;

[0053] After the input sequence z0 passes through the encoder part of the transformer, the cls label is extracted separately and put into the classifier for classification. The classification result is used as the pedestrian re-identification result.

[0054] Furthermore, during the training process, the pedestrian re-identification model uses identity loss, triplet loss, and intermediate loss to calculate the overall loss based on the classification results, and optimizes the entire model parameters through backpropagation.

[0055] Furthermore, the overall loss is calculated as follows:

[0056] The model output feature map uses identity loss and triplet loss as feature losses. In the two rounds of model training, the feature losses are calculated using the output of the second transformer block and the final transformer, respectively. The feature losses from the two rounds are then fused to calculate the overall loss, as shown in the following formula:

[0057] L=αL ID +βL tri

[0058] L 1st =L main +λ1L mid

[0059] L 2nd =L main +λ2L mid

[0060] L loss =L 1st +λL 2nd

[0061] In the formula, α, β, λ1, λ2, and λ are the learnable parameters of the model, and LID Indicating loss of identity, L tri L represents ternary loss. mid L represents the loss calculated using the output features of the second transformer block. main L represents the loss calculated using the final output features of the transformer. 1st L represents the loss obtained from the first round of model training. 2nd L represents the loss obtained from the second round of model training. loss This represents the final loss of the model.

[0062] Furthermore, the pedestrian re-identification model is trained in the following manner:

[0063] Obtain pedestrian image data, perform feature sharpening enhancement processing on the pedestrian image data, and then obtain the training set;

[0064] Model construction: Using a pure transformer as the baseline, the transformer encoder is used as the backbone network, and a feature backpropagation module is added;

[0065] The model is trained using a training set and a preset loss function to obtain a trained pedestrian re-identification model.

[0066] The second technical solution adopted in this invention is:

[0067] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a pedestrian re-identification method based on feature feedback as described above.

[0068] The third technical solution adopted in this invention is:

[0069] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a pedestrian re-identification method with feature backhaul as described above.

[0070] The fourth technical solution adopted in this invention is:

[0071] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned pedestrian re-identification method based on feature feedback.

[0072] The beneficial effects of this invention are: by focusing on sharpening pedestrian features in the foreground of the image, eliminating interference from the background and occlusion, aggregating global and local features and emphasizing the enhancement of local features, and supplementing and strengthening missing detailed information, this invention effectively improves the performance of the pedestrian re-identification network. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a structural diagram of the person re-identification model based on transformer in an embodiment of the present invention;

[0075] Figure 2 This is a flowchart of a pedestrian re-identification method based on feature feedback in an embodiment of the present invention. Detailed Implementation

[0076] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0077] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0078] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0079] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0080] The transformer described in this invention was originally proposed in the field of Natural Language Processing (NLP) to process one-dimensional sequence information. Due to its powerful representation learning and modeling capabilities, it has been gradually applied to the field of computer vision. The multi-head attention mechanism based on the transformer, because it eliminates the downsampling process, can capture long-term dependencies and thus preserve more efficient information. In the image processing domain, the transformer is called a vision transformer, which crops an image into small, uniformly sized patches, similar to segmenting an image into multiple smaller pieces to fit the transformer's input format. The patches are then processed through convolutional mapping and flattening operations, transforming each patch into a one-dimensional sequence. A self-trainable cls tag is then generated and combined with the generated one-dimensional sequence to form a complete input sequence. Since images are two-dimensional signals with rich positional information, this information is lost when flattened into one dimension. Therefore, during training, a trainable parameter with a length equal to the total number of patches plus one is set as the image's positional encoding information and superimposed on the input sequence. The input sequence then passes through a multi-layer transformer encoder module consisting of a multi-head self-attention mechanism, a feedforward network, residual connections, and layer normalization. The cls tag learns discriminative features representing the input image during the self-attention process. Finally, pedestrians are classified based on the cls tag.

[0081] While the Transformer was modified for image tasks, it wasn't fully adapted. Aside from the aforementioned positional encoding information, it offered no additional improvements for image tasks. The Transformer considers all elements in the input sequence, using a self-attention mechanism to ensure each patch acquires information from many other patches, thus forming a global feature representation. However, using only the global image doesn't allow the model to focus on local details, and the linear block method struggles to avoid local background interference and is affected by misalignment of objects. Therefore, while using ViT alone achieves good results, it lacks extraction of local information and improvements for image tasks. To address these issues, considering the multi-head self-attention mechanism in the Transformer encoder, all elements in the input sequence are treated equally, ensuring each patch acquires information from many other patches, while some local and detailed information is ignored.

[0082] Therefore, this invention proposes (1) a feature sharpening module (FS), which benefits from the amplified receptive field and implicit feature enhancement technology, captures multi-scale contextual representations to expand the receptive field, and enhances pedestrian boundary details in the image by utilizing the implicit sharpening module in the sharpening module, segmenting foreground and background information in the image, and capturing implicit local information and sharpening information. In addition, the explicit boundary enhancement module in the sharpening module learns salient features to handle background clutter, delineates pedestrians from the background, and makes the model more focused on pedestrian features. (2) A feature backpropagation module (PS), which acts as a bridge to train the model in two rounds, feeding back feature maps with detailed information from the vicinity of the upper layer output of the model to the lower layer, and aggregating global and local features at specific high layers, making up for the lack of detailed information in the transformer algorithm and improving the model's recognition accuracy. In order to make full use of the model's multiple output features, this invention also proposes (3) a fusion loss, which utilizes the intermediate output features of the transformer block and the output features of the model's two rounds of training, so that the model can converge in a more stable way.

[0083] Example 1

[0084] like Figure 1 and Figure 2 As shown, this embodiment provides a pedestrian re-identification method based on feature feedback, including the following steps:

[0085] S1. Obtain the input image and perform feature sharpening processing on the input image through the feature sharpening module;

[0086] S2. Input the processed image into the trained pedestrian re-identification model and output the pedestrian re-identification result.

[0087] The pedestrian re-identification model includes a transformer encoder and a feature backpropagation module. The transformer encoder is used to perform two rounds of training to extract global information from pedestrian images. The feature backpropagation module is used to enhance the shallow features from the first round of training, aggregate global and local features and focus on enhancing local features, and then backpropagate the feature backpropagation network for the second round of training to supplement and enhance the missing detailed information.

[0088] The feature sharpening module captures contextual information while focusing on target boundaries, highlighting pedestrian features, and shares the obtained weights with the feature backpropagation module to increase the network's attention to pedestrian regions. Before segmenting and mapping the image into a transformer input sequence, feature sharpening is performed on the image to highlight pedestrian features and focus on target boundaries. The pedestrian re-identification model shares the parameters of the feature sharpening module and undergoes two rounds of training through feature backpropagation and intermediate loss to highlight key information that contributes most to the discrimination of pedestrian features. This embodiment's method highlights pedestrian features by sharpening image data and improves the utilization of shallow features, allowing the model to focus on specific regions based on previous results to improve accuracy, effectively enhancing the performance of the pedestrian re-identification model.

[0089] In one implementation, the pedestrian re-identification model is a transformer network model. The transformer encoder is used as the backbone network for two rounds of training to extract global information from pedestrian images. A feature backpropagation module is added to enhance the local information of pedestrian images. The feature backpropagation module enhances the shallow features obtained in the first round of network training, aggregates global and local features, and focuses on enhancing local features. The feature backpropagation module is then used to train the network for the second round to supplement and enhance the missing detailed information. An intermediate loss is added as a compensation loss function to make full use of the intermediate encoder layer and the feature map obtained in the first round of training to improve model performance.

[0090] In some embodiments, step S1 further includes the following step: preprocessing the image data to convert it into an input sequence adapted to the transformer.

[0091] See Figure 1 As an optional implementation, the pedestrian re-identification model is trained through the following steps:

[0092] A1. Obtain pedestrian image data, perform feature sharpening enhancement processing on the pedestrian image data, and then obtain the training set;

[0093] A2. Model Construction: Use the ViT module as the backbone network and add a feature backpropagation module;

[0094] A3. Train the model using the training set and a preset loss function to obtain the trained transformer network model.

[0095] In this embodiment, data preprocessing includes methods commonly used in pedestrian re-identification tasks, such as randomly cropping, randomly flipping, and randomly erasing the input image, which can effectively enhance the robustness of the network.

[0096] During model training, the aim is to fully utilize image information, extract discriminative information from images, map images to feature space, map images with the same label to nearby locations, and differentiate the positions of images with different labels in feature space. During model testing, the query set and retrieval set from the pedestrian re-identification dataset are mapped to feature space according to the trained model. Based on the images in the query set, the retrieval set finds the 10 closest images to it, and the retrieval results are returned in order of distance.

[0097] See Figure 2 , Figure 2 This is a structural diagram of the pedestrian re-identification model based on transformer in this invention. This embodiment uses a pure transformer as the backbone network to extract global pedestrian information. To broaden the transformer's focus on pedestrian features and capture local information, this embodiment adds a feature sharpening module to highlight pedestrian features and a feature backpropagation module to supplement and enhance missing detailed information. A fusion loss is set to fully utilize the output features during training to find more discriminative features in areas of interest, thereby strengthening the entire network's ability to represent pedestrian features.

[0098] The following section provides a detailed explanation of each module in the pedestrian re-identification method based on feature backhaul.

[0099] (1) Feature Sharpening Module

[0100] The feature sharpening module, consisting of a deep convolutional layer, sharpening units, and a multilayer perceptron block, performs the following processing on the input image:

[0101] For an input image I∈R H×W×C Where H represents height, W represents width, and C represents the number of channels, the input image I is convolved channel-wise and point-wise using a depthwise convolutional layer with both length and width P to obtain feature I′. Image feature I′ is then input into a sharpening module for feature sharpening, and finally, multilayer perceptron processing is performed. The specific formula is as follows:

[0102] I′=DW Conv(I)

[0103] I″ = MS(I′)

[0104] X = MLP(I″)

[0105] In the formula, DW Conv(·) represents a deep convolutional layer, MS(·) represents a sharpening unit, and MLP(·) represents a multilayer perceptron operation.

[0106] The sharpening unit includes a grouped convolutional layer, a boundary enhancement module, and a regular convolutional layer, and performs the following processing steps on the input image:

[0107] First, two grouped convolutional layers with different dilation values ​​are used to extract contextual features, and the features I′ are passed to the boundary enhancement module to obtain boundary enhancement features. Finally, the multi-contextual features and boundary enhancement features are concatenated along the channel dimension, and a 1×1 convolutional layer is used to obtain the enhanced feature representation, ensuring that the output image features are invariant to the input size, thus obtaining the pedestrian sharpening image features X∈R. H×W×C The specific formula is as follows:

[0108]

[0109] I″2=BE(I′)

[0110]

[0111] In the formula, Group Conv(·) represents a grouped convolutional layer. Indicates addition, BE(·) represents a boundary enhancement module, and Conv(·) represents a regular convolutional layer. This indicates the concatenation of two feature vectors.

[0112] The boundary enhancement module first encodes the local representation of feature I′ using a deep convolutional layer, enhancing the structural information in each channel to obtain feature Z, and calculating the mean value to obtain the average response of the feature map. To highlight details, the average response is subtracted from the input feature I′ to obtain feature Y. Furthermore, the spatial average value of each channel of feature Z is calculated, and a softmax operation is used to obtain the sharpening factor s for each channel of feature map Y. The sharpening factor is multiplied by feature Y and added to feature Z to obtain the sharpened boundary enhancement feature X2. The specific formula is as follows:

[0113] Z = DW Conv(I′)

[0114] Y = I′ - C Mean(Z)

[0115] s = softmax(S Mean(Z))

[0116]

[0117] In the formula, CMean(·) represents the mean calculated along the channel dimension, SMean(·) represents the mean calculated along the spatial dimension, softmax represents the normalization operation, and ⊙ represents addition. After adding a feature sharpening module to the backbone network, compared with the VIT pedestrian re-identification network model composed of pure transformers, the map performance improved by 0.6%, and the Rank-1 performance improved by 0.4%, which is a very good improvement on the Market1501 dataset, which has been almost pushed to its limits.

[0118] (2) Pedestrian Re-identification Model

[0119] The pedestrian re-identification model performs the following processing on the image after sharpening pedestrian features:

[0120] The input image features X∈R are processed by a convolutional layer containing C′ kernels of length and width P and stride S. H×W×C Perform a linear mapping to obtain N sequences of length D. i∈1,2,...,N.

[0121]

[0122] D = H * W * C

[0123] Generate a sequence x P Learnable cls tags of equal length, and the cls tags are associated with the sequence x. P The spatial information is combined and integrated into the combination through learnable positional encoding, resulting in the input sequence z0 represented as follows:

[0124]

[0125] In the formula, x cls ∈R 1×D Indicates the cls marker, E pos ∈R (N+1)×D This is represented as a positional code.

[0126] After linear mapping, the pedestrian re-identification model inputs the input sequence z0 into the encoder part, which contains multiple transformers. The encoder part includes a multi-head self-attention module and a multi-layer perception module. The formula for calculating the output sequence is as follows:

[0127] z′ l =MSA(LN(z) l-1 ))+z l-1

[0128] z l =MLP(LN(z′) l ))+z′ l

[0129] In the formula, LN(·) represents the inter-layer normalization operation, MSA(·) represents the multi-head attention operation, and MLP(·) represents the multi-layer perception operation.

[0130] In the multi-head self-attention module, multi-head is simply a linear segmentation operation that helps the module focus on more discriminative regions without changing the underlying operation. Therefore, we will only introduce the self-attention part below. The self-attention module will process the input z... l-1 With three self-trained weight matrices W q W k W v ∈R D×D Performing matrix operations is a self-extraction of information, resulting in three pairs of input z. l-1 The representation of q,k,v∈R (N+1)×D Then, matrix multiplication is performed on the transposes of matrices q and k, which is equivalent to performing operations on different representations of the same information. This is the origin of self-attention. Based on this operation, the attention matrix attn of the entire input can be extracted. l =qk T ,attn l ∈R (N+1)×(N+1) As can be seen from the operation, the positions with high values ​​in the attention matrix are the regions of high interest from the multi-head self-attention module. Then, a matrix multiplication operation is performed between the attention matrix attn and v to calculate the pedestrian's feature values ​​based on the regions of high interest. Since the multilayer sensing operation is not modified or needed in this method, it will not be quantitatively analyzed. Simply put, the multilayer sensing operation is a linear mapping of the output of the multi-head self-attention module.

[0131] The feature backpropagation module performs the following operations on the image features output from the first round of transformer training:

[0132] The feature backpropagation module aggregates global and local features and focuses on enhancing local features. It consists of three convolutional branches, one pooling branch, a Squeeze-and-Excitation (SE) module, and a feedforward neural network (FFN).

[0133] The output feature maps s1 and s2 of the first and second transformer modules are concatenated and multiplied by the channel sharpening factor s. Then, they are divided into four components, which are processed by three convolutional branches and pooling branches, and then by the SE module and FFN operation. Finally, they are concatenated with the pedestrian sharpened image feature X and fed back to the transformer encoder for the second round of model training.

[0134]

[0135] S1, S2, S3, S4 = Split(S)

[0136]

[0137] In the formula, Split(·) represents the channel separation operation, and FFN(SE(·)) indicates that the signal passes through the SE module first, followed by the FFN network operation. After adding a feature backpropagation module to the backbone network, the map performance improved by 1.1%, and the Rank-1 performance improved by 0.9%, further optimizing the model performance.

[0138] The concatenated output features X′ are again converted into transformer input sequences by the image preprocessing module. The z0, which represents pedestrian image information, is processed by the encoder part of each transformer layer. The cls label is extracted separately and put into the classifier for classification to obtain the classification result. During the training of the pedestrian re-identification model, the overall loss is calculated based on the classification result using identity loss, triplet loss and intermediate loss, and the parameters of the entire model are optimized through backpropagation.

[0139] (3) Loss Function

[0140] During the training process, the pedestrian re-identification model uses identity loss, triplet loss and intermediate loss to calculate the overall loss based on the classification results, and optimizes the entire model parameters through backpropagation.

[0141] The model output feature map uses identity loss and triplet loss as feature losses. In the two rounds of model training, the feature losses are calculated using the second transformer block and the final transformer output, respectively. The feature losses from the two rounds are then fused to calculate the loss of the final model.

[0142] L=αL ID +βL tri

[0143] L 1st =L main +λ1L mid

[0144] L 2nd =L main +λ2L mid

[0145] L loss =L 1st +λL 2nd

[0146] In the formula, α, β, λ1, λ2, and λ are the learnable parameters of the model, and L ID Indicating loss of identity, L tri L represents ternary loss.mid L represents the loss calculated using the output features of the second transformer block. main L represents the loss calculated using the final output features of the transformer. 1st L represents the loss obtained from the first round of model training. 2nd L represents the loss obtained from the second round of model training. loss This represents the final loss of the model. Using this loss allows the model to learn more features that can represent pedestrians, enhancing the robustness of the entire network. After training with the fusion loss added to the network, the map performance improved by 0.3%, and the Rank-1 performance improved by 0.2%.

[0147] In summary, this embodiment uses a pure transformer as the backbone network and proposes a feature sharpening module and a feature backpropagation module to fully extract pedestrian features and local fine-grained features from pedestrian images. Considering that the network has not only one round of training but also multiple feature outputs, this example adds a fusion loss to utilize the intermediate output of the transformer block and the output of the model's two rounds of training, thereby enhancing the network's ability to represent pedestrian features. This method has also undergone extensive validation experiments on the standard pedestrian re-identification dataset Market1501, achieving state-of-the-art performance with a map accuracy of 90.1% and a Rank-1 accuracy of 96.0%, as shown in Table 1 below.

[0148] Table 1

[0149] methods mAP(%) Rank-1 (ms) Baseline 87.1 94.5 +FS 87.7 94.9 +FS+PS 88.8 95.8 <![CDATA[ViT+FS+PS+loss mid ]]> 90.1 96.0

[0150] Example 2

[0151] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 2 This illustrates a pedestrian re-identification method based on feature feedback.

[0152] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0153] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0154] Since this electronic device is the electronic device corresponding to the pedestrian re-identification method with feature feedback in the embodiment of the present invention, and the principle of solving the problem by this electronic device is similar to that of this method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0155] Example 3

[0156] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 2 This illustrates a pedestrian re-identification method based on feature feedback.

[0157] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0158] Since this storage medium is the storage medium corresponding to a pedestrian re-identification method with feature feedback in an embodiment of the present invention, and the principle of solving the problem by this storage medium is similar to that of this method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0159] Example 4

[0160] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of a pedestrian re-identification method based on feature feedback according to various exemplary embodiments of the present application described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0161] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0162] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0163] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A pedestrian re-identification method based on feature feedback, characterized in that, Includes the following steps: Acquire the input image and perform feature sharpening processing on it; The processed image is input into the trained pedestrian re-identification model, and the pedestrian re-identification result is output. The pedestrian re-identification model includes a transformer encoder and a feature backpropagation module. The transformer encoder is used to perform two rounds of training to extract global information from pedestrian images. The feature backpropagation module is used to enhance the shallow features from the first round of training, aggregate global and local features and focus on enhancing local features, and then backpropagate the feature backpropagation network for the second round of training to supplement and enhance the missing detailed information. The input image is processed using a feature sharpening module; The feature sharpening module includes a deep convolutional layer, a sharpening unit, and a multilayer perceptron block. The feature sharpening module works as follows: For an input image I, a deep convolutional layer is used to perform channel-wise convolution and point-wise convolution on the input image I to obtain image features I′; image features I′ are then input into a sharpening unit for feature sharpening, and finally, a multilayer perceptron operation is performed, as shown in the following formula: I′=DW Conv(I) I″ = MS(I′) X = MLP(I″) In the formula, DW Conv(·) represents a deep convolutional layer, MS(·) represents a sharpening unit, and MLP(·) represents a multilayer perceptron operation; the sharpening unit includes a grouped convolutional layer, a boundary enhancement module, and a convolutional layer; The sharpening unit operates as follows: First, grouped convolutional layers are used to extract the contextual features of image feature I′, and image feature I′ is passed to the boundary enhancement module to obtain boundary enhancement features. Finally, the contextual features and boundary enhancement features are concatenated along the channel dimension, and convolutional layers are used to obtain the enhanced feature representation to ensure that the output image features are the same size as the input, thus obtaining the pedestrian sharpening image feature X, as shown in the following formula: I″2=BE(I′) In the formula, Group Conv(·) represents a grouped convolutional layer. Indicates addition, BE(·) represents a boundary enhancement module, and Conv(·) represents a convolutional layer. This indicates the concatenation of two feature vectors; I″1 represents the context feature, and I″2 represents the boundary enhancement feature.

2. The pedestrian re-identification method based on feature feedback according to claim 1, characterized in that, The boundary enhancement module operates as follows: First, a deep convolutional layer is used to encode the local representation of image feature I′. The structural information in each channel is enhanced to obtain feature Z, and the mean value is calculated to obtain the average response of the feature map. To highlight details, the average response is subtracted from image feature I′ to obtain feature Y. Furthermore, the spatial mean value of each channel of feature Z is calculated, and a softmax operation is used to obtain the sharpening factor s for each channel of feature map Y. The sharpening factor s is multiplied by feature Y and added to feature Z to obtain the sharpened boundary enhancement feature, as shown in the following formula: Z = DW Conv(I′) Y = I′ - C Mean(Z) s = softmax(S Mean(Z)) In the formula, C Mean(·) represents the mean calculated in the channel dimension, S Mean(·) represents the mean calculated in the spatial dimension, softamx represents the normalization operation, and ⊙ represents multiplication.

3. The pedestrian re-identification method based on feature feedback according to claim 1, characterized in that, The transformer encoder is used to perform the following processing on the image after pedestrian feature sharpening: Linear mapping is performed on the input image features X to obtain N sequences x. P ; Generate a sequence x P Learnable cls tags of equal length, and the cls tags are associated with the sequence x. P The spatial information is combined and integrated into the combination through learnable positional encoding to obtain the input sequence z0; After the linear mapping is completed, the input sequence z0 is fed into four transformer modules. Each transformer module contains three transformer encoders. The encoder part includes a multi-head self-attention module and a multilayer perception module. The formula for calculating the output sequence is as follows: z′ l =MSA(LN(z l-1 ))+z l-1 With l =MLP(LN(z′ l ))+z′ l In the formula, LN(·) represents the inter-layer normalization operation, MSA(·) represents the multi-head attention operation, and MLP(·) represents the multi-layer perception operation.

4. The pedestrian re-identification method based on feature feedback according to claim 3, characterized in that, The feature backpropagation module consists of three convolutional branches, one pooling branch, a Squeeze-and-Excitation module, and a feedforward neural network. The feature backpropagation module performs the following operations on the image features output from the first round of transformer training: After concatenating the output feature maps s1 and s2 from the first and second transformer modules, multiplying them by the channel sharpening factor s, and then dividing them into four components, each processed through three convolutional branches and pooling branches, followed by Squeeze-and-Excitation modules and feedforward neural network operations, the result is concatenated with the pedestrian sharpened image feature X and fed back to the transformer encoder for the second round of model training. The formula is as follows: S1, S2, S3, S4 = Split(S) In the formula, Split(·) represents the channel separation operation, and FFN(SE(·)) represents passing through the Squeeze-and-Excitation module first, and then performing the feedforward neural network.

5. The pedestrian re-identification method based on feature feedback according to claim 4, characterized in that, After obtaining the feature X′, the feature X′ is linearly mapped to obtain the sequence x. P The cls tag is associated with the sequence x. P The spatial information is combined and integrated into the combination through learnable positional encoding to obtain the input sequence z0; After the input sequence z0 passes through the encoder part of the transformer, the cls label is extracted separately and put into the classifier for classification. The classification result is used as the pedestrian re-identification result.

6. The pedestrian re-identification method based on feature feedback according to claim 1, characterized in that, During the training process, the pedestrian re-identification model uses identity loss, triplet loss, and intermediate loss to calculate the overall loss based on the classification results, and optimizes the entire model parameters through backpropagation.

7. The pedestrian re-identification method based on feature feedback according to claim 6, characterized in that, The overall loss is calculated as follows: The model output feature map uses identity loss and triplet loss as feature losses. In the two rounds of model training, the feature losses are calculated using the output of the second transformer block and the final transformer, respectively. The feature losses from the two rounds are then fused to calculate the overall loss, as shown in the following formula: L=αL ID +βL tri L 1st =L main +λ1L mid L 2nd =L main +λ2L mid THE loss =L 1st +λL 2nd In the formula, α, β, λ1, λ2, and λ are the learnable parameters of the model, and L ID Indicating loss of identity, L tri L represents ternary loss. mid L represents the loss calculated using the output features of the second transformer block. main L represents the loss calculated using the final output features of the transformer. 1st L represents the loss obtained from the first round of model training. 2nd L represents the loss obtained from the second round of model training. loss This represents the final loss of the model.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the method as described in any one of claims 1 to 7.

Images