Feature return pedestrian re-identification method and electronic equipment

Abstract

The invention discloses a pedestrian re-identification method based on feature return and electronic equipment, and the method comprises the steps: obtaining an input image, and carrying out the feature sharpening of the input image; inputting the processed image into the trained pedestrian re-identification model, and outputting a pedestrian re-identification result; the pedestrian re-identification model comprises a transform encoder and a feature return module; the encoder is used for carrying out two rounds of training to extract global information of the pedestrian image; and the feature return module is used for enhancing the shallow features of the first round of training, aggregating global and local features, emphatically enhancing the local features, and returning the features back to the network for second round of training so as to supplement and enhance missing detailed information. According to the method, pedestrian features are highlighted by sharpening image data, utilization of shallow features is improved, the model is allowed to pay attention to a specific area based on a previous result to improve accuracy, the performance of a pedestrian re-recognition model is effectively improved, and the method can be widely applied to the technical field of image recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision image recognition, and in particular to a pedestrian re-identification method and electronic equipment with feature feedback. Background Art

[0002] Person Re-identification (Re-ID) is an important research direction in the field of computer vision, which aims to identify and match the same pedestrian under different cameras. With the popularization of intelligent monitoring systems and the surge in data volume, the application of person re-identification in security monitoring, public safety, intelligent transportation, business analysis and human-computer interaction is becoming increasingly important.

[0003] Person re-identification is a challenging task. Changes in illumination, perspective, background clutter, and the diversity of a person's appearance (such as clothing, posture, body shape, etc.) can cause the same person to appear very different under different cameras. Effectively solving these problems is crucial to improving the intelligence level of the system. In order to solve these problems, researchers have begun to explore various technologies and algorithms to improve the accuracy and robustness of re-identification.

[0004] In recent years, the research on pedestrian re-identification has made significant progress. At present, deep learning-based methods dominate the field of pedestrian re-identification. It is necessary to train a deep model for pedestrian re-identification to extract pedestrian features for classification. Training a deep learning model for pedestrian re-identification can be divided into the following steps. First, a set of pedestrian images in the training set are processed and input into the deep neural network. After a large number of calculations, a set of feature vectors representing pedestrians are obtained. The vectors are then substituted into the designed loss function to calculate the loss value, and then the model parameters are updated through gradient backpropagation and optimizer. Depending on the focus, researchers will choose to study from four aspects: data sets, representation learning, metric learning, and optimization sorting. Among them, metric learning as the feature extraction part is the most studied, and it is crucial to design a network model that effectively extracts pedestrian features.

[0005] Person re-identification is mainly studied from the perspectives of global features, local features, auxiliary features, and attention features: (1) Person re-identification methods based on global images are the main research methods in the early stage and are relatively simple. The image is directly input into the CNN network for convolution to obtain global features of the pedestrian's clothing, body shape, posture, etc., and then classified. This approach achieved good results at the beginning, but in the inevitable downsampling process, the resolution will gradually decrease and irreversibly, making it easy for the network to ignore local discriminative features. In addition, directly extracting features from the entire image is easily interfered by occlusion, background, and human posture, thus affecting the recognition rate in these situations. (2) Person re-identification methods based on local features divide the image into regions or estimate the posture to extract feature information, prompting the network to pay more attention to local regions. The network can use local information for matching and classification, and has achieved good results. However, under severe occlusion, even local information contains a lot of interference noise, and is powerless for misaligned people. (3) Pedestrian re-identification methods based on auxiliary features use some auxiliary information such as semantic information, camera information, GAN, information enhancement, etc. to enhance feature information. This strategy significantly improves the recognition accuracy, but due to the introduction of additional information, it will inevitably lead to a large amount of computing power consumption and the final recognition rate is highly dependent on the cleanliness of the pedestrian information extracted in the first stage. There is a problem that the discriminative fine-grained features of pedestrians are regarded as interference and classified as background information, resulting in the loss of key pedestrian information. (4) Pedestrian re-identification methods based on attention mechanisms. There are many attention mechanisms based on convolution operations that can mine the most discriminative areas in pedestrian images. However, due to the limitation of the convolution kernel to the receptive field and the loss of resolution caused by downsampling, it can usually only focus on a small area and cannot solve the problem of long-term dependence. Summary of the invention

[0006] In order to solve at least one of the technical problems existing in the prior art to a certain extent, an object of the present invention is to provide a pedestrian re-identification method, electronic device and medium with feature feedback.

[0007] The first technical solution adopted by the present invention is:

[0008] A pedestrian re-identification method based on feature feedback includes the following steps:

[0009] Obtain an input image, and perform feature sharpening processing on the input image;

[0010] Input the processed image into the trained person re-identification model and output the person re-identification result;

[0011] Among them, the pedestrian re-identification model includes a transformer encoder and a feature feedback module; the transformer encoder is used to perform two rounds of training to extract global information of pedestrian images; the feature feedback module is used to enhance the shallow features of the first round of training, aggregate global and local features and focus on enhancing local features, and the feature feedback network is used for a second round of training to supplement and strengthen the missing detailed information.

[0012] Further, a feature sharpening module is used to perform feature sharpening processing on the input image;

[0013] The feature sharpening module includes a deep convolution layer, a sharpening unit and a multi-layer perception block;

[0014] The feature sharpening module works as follows:

[0015] For an input image I∈R H×W×C , where H represents height, W represents width, and C represents the number of channels. A deep convolutional layer with length and width P is used to perform channel-by-channel convolution and point-by-point convolution on the input image I to obtain image features I′; the image features I′ are input into the sharpening unit for feature sharpening, and finally a multi-layer perception operation is performed. The formula is as follows:

[0016] I′=DWConv(I)

[0017] I″=MS(I′)

[0018] X=MLP(I″)

[0019] Where DWConv(·) represents the deep convolutional layer, MS(·) represents the sharpening unit, and MLP(·) represents the multi-layer perception operation.

[0020] Further, the sharpening unit includes a grouped convolutional layer, a boundary enhancement module and a common convolutional layer;

[0021] The sharpening unit works as follows:

[0022] First, two grouped convolutional layers with different dilation values ​​are used to extract the contextual features of image feature i′, and the image feature I′ is passed to the boundary enhancement module to obtain the boundary enhancement feature; finally, the contextual features and boundary enhancement features are connected along the channel dimension, and a 1×1 convolutional layer is used to obtain the enhanced feature representation to ensure that the output image features are the same size as the input, and the pedestrian sharpened image feature X∈R is obtained. H×W×C , the formula is as follows:

[0023]

[0024] I″2=BE(I′)

[0025]

[0026] In the formula, Group Conv(·) represents the group convolution layer, represents addition, BE(·) represents the boundary enhancement module, Conv(·) represents the ordinary convolutional layer, Represents the concatenation of two feature vectors; I″1 represents the context feature, and I″2 represents the boundary enhancement feature.

[0027] Furthermore, the boundary enhancement module works as follows:

[0028] First, the local representation of the image feature I′ is encoded using a deep convolutional layer, and the structural information in each channel is enhanced to obtain feature Z, and the average response of the feature map is calculated. In order to highlight the details, the average response is subtracted from the image feature I′ to obtain feature Y. In addition, the spatial average of each channel of feature Z is calculated, and the softmax operation is used to obtain the sharpening factor s of each channel of feature map Y. The sharpening factor s is multiplied by feature Y and added to feature Z to obtain the sharpened boundary enhancement feature, which is expressed as follows:

[0029] Z=DW Conv(I′)

[0030] Y=I′-C Mean(Z)

[0031] s=softmax(S Mean(Z))

[0032]

[0033] Where C Mean(·) represents the mean calculated in the channel dimension, S Mean(·) represents the mean calculated in the spatial dimension, softmax represents the normalization operation, and ⊙ represents addition.

[0034] Furthermore, the transformer encoder is used to perform the following processing on the image after the pedestrian feature is sharpened:

[0035] Use a convolutional layer containing C′ convolution kernels with length and width of P and stride of S to input image features X∈R H×W×C Perform linear mapping to obtain N sequences of length D i∈1,2,...,N, where:

[0036]

[0037] D=H*W*C

[0038] Generate a sequence x PA learnable cls tag of equal length, which is combined with the sequence x P Combination, and integrate spatial information into the combination through learnable position encoding, to obtain the input sequence z0, the formula is as follows:

[0039]

[0040] In the formula, x cls ∈R 1×D Indicates the cls tag, E pos ∈R (N+1)×D Represented as positional encoding;

[0041] After the linear mapping is completed, the input sequence z0 is input into 4 transformer modules. Each transformer module contains 3 transformer encoders. The encoder part contains a multi-head self-attention module and a multi-layer perception module. The formula for calculating the output sequence is as follows:

[0042] z′ l =MSA(LN(z l-1 ))+z l-1

[0043] z l =MLP(LN(z′ l ))+z′ l

[0044] Where LN(·) represents the inter-layer normalization operation, MSA(·) represents the multi-head attention operation, and MLP(·) represents the multi-layer perception operation.

[0045] Furthermore, the feature feedback module is composed of three convolution branches, a pooling branch, a Squeeze-and-Excitation (SE) module and a feed-forward neural network (FFN);

[0046] The feature feedback module performs the following operations on the image features output by the first round of transformer training:

[0047] After the output feature maps s1 and s2 of the first and second transformer modules are concatenated, they are multiplied by the channel sharpening factor s, and then divided into four components and passed through three convolution branches and pooling branches respectively. After passing through the Squeeze-and-Excitation module and the feedforward neural network operation, they are concatenated with the pedestrian sharpened image feature X and then sent back to the transformer encoder for the second round of model training. The formula is as follows:

[0048]

[0049] S1,S2,S3,S4=Split(S)

[0050]

[0051] Where Split(·) represents the channel separation operation, and FFN(SE(·)) means that the Squeeze-and-Excitation module is first used, followed by a feedforward neural network.

[0052] Furthermore, after obtaining the feature X′, the feature X′ is linearly mapped to obtain the sequence x P , associate the cls tag with the sequence x P Combine and integrate spatial information into the combination through learnable position encoding to obtain the input sequence z0;

[0053] After the input sequence z0 passes through the encoder part of the transformer, the cls tag will be extracted separately and put into the classifier for classification to obtain the classification result as the pedestrian re-identification result.

[0054] Furthermore, during the training process of the pedestrian re-identification model, the identity loss, triplet loss and intermediate loss are used to calculate the overall loss according to the classification results, and the entire model parameters are optimized through back propagation.

[0055] Furthermore, the overall loss is calculated as follows:

[0056] The model output feature map uses identity loss and triplet loss as feature loss. In two rounds of model training, the second transformer block and the final transformer output are used to calculate the feature loss respectively. The feature losses of the two rounds are fused to calculate the overall loss. The formula is as follows:

[0057] L=αL ID +βL tri

[0058] L 1st =L main +λ1L mid

[0059] L 2nd =L main +λ2L mid

[0060] L loss =L 1st +λL 2nd

[0061] Where α, β, λ1, λ2, λ are the learnable parameters of the model, and LID represents identity loss, L tri represents the ternary loss, L mid represents the loss calculated using the output features of the second transformer block, L main Represents the loss calculated using the final output feature of the transformer, L 1st represents the loss obtained from the first round of model training, L 2nd represents the loss obtained from the second round of model training, L loss Represents the final loss of the model.

[0062] Furthermore, the pedestrian re-identification model is trained in the following way:

[0063] Obtain pedestrian image data, perform feature sharpening and enhancement processing on the pedestrian image data, and then obtain a training set;

[0064] Model construction: Use pure transformer as the baseline, use transformer encoder as the backbone network, and add a feature feedback module;

[0065] The model is trained using the training set and the preset loss function to obtain a trained person re-identification model.

[0066] The second technical solution adopted by the present invention is:

[0067] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a pedestrian re-identification method with feature feedback as described above.

[0068] The third technical solution adopted by the present invention is:

[0069] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a pedestrian re-identification method with feature feedback as described above.

[0070] The fourth technical solution adopted by the present invention is:

[0071] A computer program product or a computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned pedestrian re-identification method with feature feedback.

[0072] The beneficial effects of the present invention are as follows: the present invention sharpens pedestrian features by focusing on the foreground of the image, eliminates interference from the background and occlusion, aggregates global and local features and focuses on enhancing local features, supplements and strengthens missing detailed information, and effectively improves the performance of the pedestrian re-identification network. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0074] Figure 1 is a structural diagram of a transformer-based person re-identification model in an embodiment of the present invention;

[0075] Figure 2 It is a flow chart of a pedestrian re-identification method with feature feedback in an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0077] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0078] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0079] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0080] The transformer described in the present invention was first proposed in the field of natural language processing (NLP). It was originally used to process one-dimensional sequence information. Due to its powerful representation learning and modeling capabilities, it has been gradually applied in the field of computer vision. The transformer-based multi-head attention mechanism can capture long-term dependencies without the downsampling process, so it can save more effective information. The application of transformer in the field of images is called vision transformer, which crops the image into small blocks of the same size, namely patches, which is similar to dividing the image into multiple small blocks to adapt to the input format of the transformer. The patches are then mapped and flattened through convolution operations, so that each patch becomes a one-dimensional sequence information, and then a self-trainable CLS tag is generated and combined with the generated one-dimensional sequence to form a complete input sequence. Since the image is a two-dimensional signal with rich position information, this part of information will be lost after flattening into one dimension. Therefore, a trainable parameter with a length of the total number of patches plus one will be set during the training process as the image position encoding information superimposed on the input sequence. Then the input sequence passes through multiple layers of transformer encoder modules consisting of a multi-head self-attention mechanism, a feedforward network, a residual connection and layer normalization. The CLS tag will learn the discriminative features representing the input image during the self-attention process, and finally the pedestrians will be classified according to the CLS tag.

[0081] Although the transformer has been modified for image tasks, it is not fully adapted to image tasks. In addition to the above-mentioned position encoding information, there is no additional improvement for image tasks. The transformer considers all elements in the input sequence, and through the self-attention mechanism, each patch obtains the information of many other patches, thereby forming a global feature representation. Using only the global image cannot make the model pay attention to local detailed information, and using the linear block method, it is difficult to avoid local background interference and is affected by the misalignment of the characters. It can be seen that although the method of using only ViT has achieved good results, it lacks the extraction of local information and the improvement of image tasks. To solve the above problems, considering the multi-head self-attention mechanism in the transformer encoder part, all elements in the input sequence are treated equally, so that each patch obtains the information of many other patches, while some local information and detailed information are ignored.

[0082] Therefore, the present invention proposes (1) a feature sharpening module (FS), which benefits from the enlarged receptive field and implicit feature enhancement technology, captures multi-scale contextual representation to expand the receptive field, and enhances the pedestrian boundary details in the image by utilizing the implicit sharpening module in the sharpening module, segmenting the foreground and background information in the image, and capturing implicit local information and sharpening information. In addition, the explicit boundary enhancement module in the sharpening module learns salient features to deal with background clutter, delineates pedestrians from the background, and enables the model to pay more attention to pedestrian features. (2) A feature feedback module (PS), which acts as a bridge to train the model for two rounds, feeds back feature maps with detailed information from the upper layer output of the model to the lower layer, and aggregates global and local features at specific high levels, making up for the shortcomings of the transformer algorithm that lacks detailed information and improving the model recognition accuracy. In order to make full use of the multiple output features of the model, the present invention also proposes (3) a fusion loss, which utilizes the intermediate output features of the transformer block and the output features of the two rounds of model training, so that the model can converge in a more stable manner.

[0083] Example 1

[0084] like Figure 1 and Figure 2 As shown, this embodiment provides a pedestrian re-identification method based on feature feedback, including the following steps:

[0085] S1, obtaining an input image, and performing feature sharpening processing on the input image through a feature sharpening module;

[0086] S2, input the processed image into the trained pedestrian re-identification model, and output the pedestrian re-identification result;

[0087] Among them, the pedestrian re-identification model includes a transformer encoder and a feature feedback module; the transformer encoder is used to perform two rounds of training to extract global information of pedestrian images; the feature feedback module is used to enhance the shallow features of the first round of training, aggregate global and local features and focus on enhancing local features, and the feature feedback network is used for a second round of training to supplement and strengthen the missing detailed information.

[0088] The feature sharpening module focuses on the target boundary while capturing contextual information, highlighting the pedestrian features, and shares the obtained weights with the feature feedback module to increase the network's attention to the pedestrian area. Before encoding the image segmentation map into a transformer input sequence, the image is first feature sharpened to highlight the pedestrian features in order to focus on the target boundary; the pedestrian re-identification model shares the feature sharpening module parameters, performs two rounds of training through feature feedback and intermediate loss, and highlights the key information that contributes most to the discriminant row features. The method of this embodiment highlights pedestrian features by sharpening image data and improves the utilization of shallow features, allowing the model to focus on specific areas based on previous results to improve accuracy, effectively improving the performance of the pedestrian re-identification model.

[0089] As an implementation method, the pedestrian re-identification model is a transformer network model, and the transformer encoder is used as the backbone network for two rounds of training to extract the global information of the pedestrian image; a feature feedback module is added to enhance the local information of the pedestrian image. The feature feedback module enhances the shallow features of the first round of network training, aggregates global and local features and focuses on enhancing local features. The feedback network is trained in the second round to supplement and strengthen the missing detailed information; an intermediate loss is added as a compensation loss function to make full use of the intermediate encoder layer and the feature map obtained in the first round of training to improve model performance.

[0090] In some embodiments, step S1 also includes the following steps: preprocessing the image data to convert it into an input sequence suitable for the transformer.

[0091] See also Figure 1 As an optional implementation, the pedestrian re-identification model is trained by the following steps:

[0092] A1. Obtain pedestrian image data, perform feature sharpening and enhancement processing on the pedestrian image data, and obtain a training set;

[0093] A2. Build the model: Use the ViT module as the backbone network and add a feature feedback module;

[0094] A3. Use the training set and the preset loss function to train the model to obtain the trained transformer network model.

[0095] In this embodiment, data preprocessing includes random cropping, random horizontal flipping, and random erasing of input images, which are commonly used in pedestrian re-identification tasks and can effectively enhance the robustness of the network.

[0096] In the model training phase, the goal is to make full use of image information, extract discriminative information from the image, map the image to the feature space, map images with the same label to similar positions, and spread the positions of images with different labels in the feature space. In the model testing phase, the query set and retrieval set in the person re-identification dataset are mapped to the feature space according to the trained model, and the 10 closest images to the image in the query set are found in the retrieval set, and the retrieval results are returned according to the distance sorting.

[0097] See also Figure 2 , Figure 2 This is a structural diagram of the transformer-based pedestrian re-identification model of the present invention. This embodiment uses a pure transformer as the backbone network to extract the global information of pedestrians. In order to expand the transformer's attention to pedestrian features and its ability to capture local information, this embodiment adds a feature sharpening module to highlight pedestrian features and adds a feature feedback module to supplement and strengthen missing detailed information. The fusion loss is set to make full use of the output features in the training process to find more discriminative features of the low-attention area, thereby enhancing the ability of the entire network to characterize pedestrian features.

[0098] The following is a detailed explanation of each module in the pedestrian re-identification method based on feature feedback.

[0099] (1) Feature sharpening module

[0100] The feature sharpening module deep convolutional layer, sharpening unit and multi-layer perception block perform the following processing on the input image:

[0101] For an input image I∈R H×W×C , where H represents height, W represents width, and C represents the number of channels. The input image I is convolved channel by channel and point by point using a deep convolutional layer with length and width P to obtain feature I′. The image feature I′ is input into the sharpening module for feature sharpening, and finally a multi-layer perception operation is performed. The specific formula is as follows:

[0102] I′=DW Conv(I)

[0103] I″=MS(I′)

[0104] X=MLP(I″)

[0105] Where DW Conv(·) represents the deep convolutional layer, MS(·) represents the sharpening unit, and MLP(·) represents the multi-layer perception operation.

[0106] The sharpening unit includes a grouped convolution layer, a boundary enhancement module and a common convolution layer, and performs the following processing steps on the input image:

[0107] First, two grouped convolutional layers with different dilation values ​​are used to extract context features, and the feature I′ is passed to the boundary enhancement module to obtain the boundary enhancement feature. Finally, the multi-context features and boundary enhancement features are concatenated along the channel dimension, and a 1×1 convolutional layer is used to obtain the enhanced feature representation, ensuring that the output image features are the same size as the input to obtain the pedestrian sharpened image feature X∈R H×W×C The specific formula is as follows:

[0108]

[0109] I″2=BE(I′)

[0110]

[0111] Where Group Conv(·) represents the group convolution layer, represents addition, BE(·) represents the boundary enhancement module, Conv(·) represents the ordinary convolutional layer, Represents the concatenation of two feature vectors.

[0112] The boundary enhancement module first uses a deep convolutional layer to encode the local representation of feature I', enhances the structural information in each channel to obtain feature Z, and calculates the average value to obtain the average response of the feature map. In order to highlight the details, the average response is subtracted from the input feature I' to obtain feature Y. In addition, the spatial average of each channel of feature Z is calculated, and the softmax operation is used to obtain the sharpening factor s of each channel of feature map Y. The sharpening factor is multiplied by feature Y and added to feature Z to obtain the sharpened boundary enhancement feature X2. The specific formula is as follows:

[0113] Z=DW Conv(I′)

[0114] Y=I′-C Mean(Z)

[0115] s=softmax(S Mean(Z))

[0116]

[0117] Where C Mean(·) represents the mean calculated in the channel dimension, S Mean(·) represents the mean calculated in the spatial dimension, softmax represents the normalization operation, and ⊙ represents addition. After adding the feature sharpening module to the backbone network, compared with the pure transformer-based person re-identification network VIT model, the map is improved by 0.6% and the Rank-1 is improved by 0.4%, which is a very good improvement on the Market1501 dataset that has been almost developed to the limit.

[0118] (2) Person Re-ID Model

[0119] The pedestrian re-identification model performs the following processing on the image after pedestrian feature sharpening:

[0120] Use a convolutional layer containing C′ convolution kernels with length and width of P and stride of S to input image features X∈R H×W×C Perform linear mapping to obtain N sequences of length D i∈1,2,...,N.

[0121]

[0122] D=H*W*C

[0123] Generate a sequence x P A learnable cls tag of equal length, which is combined with the sequence x P Combination, and integrating spatial information into the combination through learnable position encoding, the input sequence z0 is expressed as follows:

[0124]

[0125] In the formula, x cls ∈R 1×D Indicates the cls tag, E pos ∈R (N+1)×D Represented as a positional encoding.

[0126] After the linear mapping is completed, the pedestrian re-identification model inputs the input sequence z0 into the encoder part containing multiple transformers. The encoder part contains a multi-head self-attention module and a multi-layer perception module. The formula for calculating the output sequence is as follows:

[0127] z′ l =MSA(LN(z l-1 ))+z l-1

[0128] z l =MLP(LN(z′ l ))+z′ l

[0129] Where LN(·) represents the inter-layer normalization operation, MSA(·) represents the multi-head attention operation, and MLP(·) represents the multi-layer perception operation.

[0130] In the multi-head self-attention module, the multi-head is just a linear segmentation operation, which helps the module to pay attention to more discriminative areas and does not change the actual operation. Therefore, only the self-attention part will be introduced below. The self-attention module will input z l-1 With three self-trained weight matrices W q ,W k ,W v ∈R D×D Perform matrix operations, which is a self-extraction of information, and obtain three pairs of input z l-1 Representation q,k,v∈R (N+1)×D , and then perform matrix multiplication on the transpose of matrix q and matrix k, which is equivalent to performing operations on different representations of the same information. This is the origin of self-attention. According to this step, the attention matrix attn of the entire input can be extracted l =qk T ,attn l ∈R (N+1)×(N+1) According to the operation, the position with high value in the attention matrix is ​​the area where the multi-head self-attention module pays high attention. Then, the attention matrix attn is multiplied with v to calculate the pedestrian's feature value based on the high attention area. Since the multi-layer perception operation will not be changed or required in this method, no quantitative analysis is performed. In short, the multi-layer perception operation is to linearly map the output of the multi-head self-attention module.

[0131] The feature feedback module performs the following operations on the image features output by the first round of transformer training:

[0132] The feature feedback module aggregates global and local features and focuses on enhancing local features. It consists of three convolution branches, a pooling branch, a Squeeze-and-Excitation (SE) module and a feedforward neural network (FFN).

[0133] The output feature maps s1 and s2 of the first and second transformer modules are concatenated and multiplied with the channel sharpening factor s, then divided into four components and passed through three convolution branches and pooling branches, and then through the SE module and FFN operation. After that, they are concatenated with the pedestrian sharpened image feature X and returned to the transformer encoder for the second round of model training.

[0134]

[0135] S1,S2,S3,S4=Split(S)

[0136]

[0137] Where Split(·) represents the channel separation operation, and FFN(SE(·)) means that the SE module is first used, and then the FFN network operation is performed. After adding the feature feedback module to the backbone network, the map is improved by 1.1% and the Rank-1 is improved by 0.9%, which further optimizes the model performance.

[0138] The spliced ​​output feature X′ is converted into a transformer input sequence again through the image preprocessing module. After z0 representing the pedestrian image information passes through the encoder part of the transformer layer by layer, the cls tag will be extracted separately and put into the classifier for classification to obtain the classification result. In the process of training the pedestrian re-identification model, the identity loss, triplet loss and intermediate loss are used to calculate the overall loss according to the classification results, and the entire model parameters are optimized through back propagation.

[0139] (3) Loss Function

[0140] During the training process of the pedestrian re-identification model, the identity loss, triplet loss and intermediate loss are used to calculate the overall loss according to the classification results, and the entire model parameters are optimized through back propagation.

[0141] The model output feature map uses identity loss and triplet loss as feature loss. In two rounds of model training, the second transformer block and the final transformer output are used to calculate the feature loss respectively, and the feature losses of the two rounds are fused to calculate the loss of the final model.

[0142] L=αL ID +βL tri

[0143] L 1st =L main +λ1L mid

[0144] L 2nd =L main +λ2L mid

[0145] L loss =L 1st +λL 2nd

[0146] Where α, β, λ1, λ2, λ are the learnable parameters of the model, and L ID represents identity loss, L tri represents the ternary loss, Lmid represents the loss calculated using the output features of the second transformer block, L main Represents the loss calculated using the final output feature of the transformer, L 1st represents the loss obtained from the first round of model training, L 2nd represents the loss obtained from the second round of model training, L loss Represents the final loss of the model. After using this loss, the model learns more features that can characterize pedestrians, enhancing the robustness of the entire network. After adding fusion loss to the network for training, map increased by 0.3% and Rank-1 increased by 0.2%.

[0147] In summary, this embodiment uses pure transformer as the skeleton network, proposes a feature sharpening module and a feature feedback module, and fully extracts pedestrian features and local fine-grained features in pedestrian images. Considering that the network has not only one round of training but also multiple feature outputs, this example adds a fusion loss to utilize the intermediate output of the transformer block and the output of two rounds of model training, thereby enhancing the network's ability to characterize pedestrian features. This method has also conducted a large number of verification experiments in the standard dataset Market1501 for pedestrian re-identification, and achieved the most advanced performance level, reaching 90.1% of the map and 96.0% of the Rank-1, as shown in Table 1 below.

[0148] Table 1

[0149] methods mAP(%) Rank-1(ms) Baseline 87.1 94.5 +FS 87.7 94.9 +FS+PS 88.8 95.8 <![CDATA[ViT+FS+PS+loss mid ]]> 90.1 96.0

[0150] Example 2

[0151] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 2 A pedestrian re-identification method with feature feedback is shown.

[0152] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0153] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0154] Since the electronic device is an electronic device corresponding to the pedestrian re-identification method based on feature feedback in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0155] Example 3

[0156] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 2 A pedestrian re-identification method with feature feedback is shown.

[0157] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0158] Since the storage medium is a storage medium corresponding to a feature feedback pedestrian re-identification method of an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0159] Example 4

[0160] In some possible implementations, various aspects of the method of the embodiments of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a pedestrian re-identification method for feature feedback according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing various embodiments may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0161] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0162] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0163] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A pedestrian re-identification method based on feature feedback, characterized in that: The following steps are involved: Obtain an input image, and perform feature sharpening processing on the input image; Input the processed image into the trained person re-identification model and output the person re-identification result; Among them, the pedestrian re-identification model includes a transformer encoder and a feature feedback module; the transformer encoder is used to perform two rounds of training to extract global information of pedestrian images; the feature feedback module is used to enhance the shallow features of the first round of training, aggregate global and local features and focus on enhancing local features, and the feature feedback network is used for a second round of training to supplement and strengthen the missing detailed information.

2. The method for pedestrian re-identification by feature feedback according to claim 1, characterized in that: The feature sharpening module is used to perform feature sharpening processing on the input image; The feature sharpening module includes a deep convolution layer, a sharpening unit and a multi-layer perception block; The feature sharpening module works as follows: For an input image I, a deep convolution layer is used to perform channel-by-channel convolution and point-by-point convolution on the input image I to obtain the image feature I′; the image feature I′ is input into the sharpening unit for feature sharpening, and finally a multi-layer perception operation is performed. The formula is expressed as follows: I′=DWConv(I) I″=MS(I′) X=MLP(I″) Where DW Conv(·) represents the deep convolutional layer, MS(·) represents the sharpening unit, and MLP(·) represents the multi-layer perception operation.

3. The method for pedestrian re-identification by feature feedback according to claim 2, characterized in that: The sharpening unit includes a grouped convolutional layer, a boundary enhancement module and a convolutional layer; The sharpening unit works as follows: First, the grouped convolutional layer is used to extract the contextual features of the image feature I′, and the image feature I′ is passed to the boundary enhancement module to obtain the boundary enhancement feature; finally, the contextual features and boundary enhancement features are connected along the channel dimension, and the convolutional layer is used to obtain the enhanced feature representation to ensure that the output image features and the input size remain unchanged, and the pedestrian sharpened image feature X is obtained. The formula is as follows: I″2=BE(I′) In the formula, Group Conv(·) represents the group convolution layer, ⊕ represents addition, BE(·) represents the boundary enhancement module, Conv(·) represents the convolution layer, Represents the concatenation of two feature vectors; I″1 represents the context feature, and I″2 represents the boundary enhancement feature.

4. The method for pedestrian re-identification by feature feedback according to claim 3, characterized in that: The boundary enhancement module works as follows: First, the local representation of the image feature I′ is encoded using a deep convolutional layer, and the structural information in each channel is enhanced to obtain feature Z, and the average response of the feature map is calculated. In order to highlight the details, the average response is subtracted from the image feature I′ to obtain feature Y. In addition, the spatial average of each channel of feature Z is calculated, and the softmax operation is used to obtain the sharpening factor s of each channel of feature map Y. The sharpening factor s is multiplied by feature Y and added to feature Z to obtain the sharpened boundary enhancement feature, which is expressed as follows: Z=DW Conv(I′) Y=I′-C Mean(Z) s=softmax(S Mean(Z)) Where C Mean(·) represents the mean calculated in the channel dimension, S Mean(·) represents the mean calculated in the spatial dimension, softmax represents the normalization operation, and ⊙ represents addition.

5. The method for pedestrian re-identification by feature feedback according to claim 1, characterized in that: The transformer encoder is used to perform the following processing on the image after the pedestrian feature is sharpened: Linearly map the input image feature X to obtain N sequences x P ; Generate a sequence x P A learnable cls tag of equal length, which is combined with the sequence x P Combine and integrate spatial information into the combination through learnable position encoding to obtain the input sequence z0; After the linear mapping is completed, the input sequence z0 is input into 4 transformer modules. Each transformer module contains 3 transformer encoders. The encoder part contains a multi-head self-attention module and a multi-layer perception module. The formula for calculating the output sequence is as follows: z′ l =MSA(LN(z l-1 ))+z l-1 With l =MLP(LN(z′ l ))+z′ l Where LN(·) represents the inter-layer normalization operation, MSA(·) represents the multi-head attention operation, and MLP(·) represents the multi-layer perception operation.

6. The method for pedestrian re-identification by feature feedback according to claim 5, characterized in that: The feature feedback module consists of three convolution branches, a pooling branch, a Squeeze-and-Excitation module and a feedforward neural network; The feature feedback module performs the following operations on the image features output by the first round of transformer training: After the output feature maps s1 and s2 of the first and second transformer modules are concatenated, they are multiplied by the channel sharpening factor s, and then divided into four components and passed through three convolution branches and pooling branches respectively. After passing through the Squeeze-and-Excitation module and the feedforward neural network operation, they are concatenated with the pedestrian sharpened image feature X and then sent back to the transformer encoder for the second round of model training. The formula is as follows: S1,S2,S3,S4=Split(S) X′=FFN(SE(S′))⊕X Where Split(·) represents the channel separation operation, and FFN(SE(·)) means that the Squeeze-and-Excitation module is first used, followed by a feedforward neural network.

7. The method for pedestrian re-identification by feature feedback according to claim 6, characterized in that: After obtaining the feature X′, perform linear mapping on the feature X′ to obtain the sequence x P , associate the cls tag with the sequence x P Combine and integrate spatial information into the combination through learnable position encoding to obtain the input sequence z0; After the input sequence z0 passes through the encoder part of the transformer, the cls tag will be extracted separately and put into the classifier for classification to obtain the classification result as the pedestrian re-identification result.

8. The method for pedestrian re-identification by feature feedback according to claim 1, characterized in that: During the training process of the pedestrian re-identification model, the identity loss, triplet loss and intermediate loss are used to calculate the overall loss according to the classification results, and the entire model parameters are optimized through back propagation.

9. The method for pedestrian re-identification by feature feedback according to claim 8, characterized in that: The overall loss is calculated as follows: The model output feature map uses identity loss and triplet loss as feature loss. In two rounds of model training, the second transformer block and the final transformer output are used to calculate the feature loss respectively. The feature losses of the two rounds are fused to calculate the overall loss. The formula is as follows: L=αL ID +βL tri L 1st =L main +λ1L mid L 2nd =L main +λ2L mid THE loss =L 1st +λL 2nd Where α, β, λ1, λ2, λ are the learnable parameters of the model, and L ID represents identity loss, L tri represents the ternary loss, L mid represents the loss calculated using the output features of the second transformer block, L main Represents the loss calculated using the final output feature of the transformer, L 1st represents the loss obtained from the first round of model training, L 2nd represents the loss obtained from the second round of model training, L loss Represents the final loss of the model.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 9.

Images