Pedestrian re-identification method and training method for optimizing fine-grained feature fusion

Abstract

The invention belongs to the technical field of computer vision and artificial intelligence, and provides a pedestrian re-identification method for optimizing fine-grained feature fusion, which comprises the following steps: converting an image to be detected into a flattening sequence, and inputting the flattening sequence to a pedestrian re-identification model to obtain pedestrian part features; performing re-identification processing on the pedestrian part features to obtain a pedestrian re-identification result; wherein the trained pedestrian re-identification model comprises a self-adaptive multi-granularity feature fusion encoder which comprises a plurality of layers of Transform encoding modules which are arranged in a stacked manner and more than one multi-granularity feature fusion module; the decoder is used for decoding the pedestrian features to obtain pedestrian part features, the pedestrian part features comprise features of p parts of the pedestrian, fusion of features of different granularities is optimized, and the model performance is improved. The invention also provides a pedestrian re-identification model training method, which promotes the student decoder to absorb the knowledge of the teacher decoder.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a pedestrian re-identification method and a training method for optimizing fine-grained feature fusion. Background Art

[0002] Person re-identification (ReID) aims to match target person images in different spatiotemporal scenarios. In recent years, the development of deep learning and the emergence of large-scale datasets have promoted significant progress in the field of ReID. However, in occluded scenarios, due to the randomness of pedestrian occlusion positions, robust pedestrian feature fusion still faces challenges. In actual monitoring systems, pedestrians are often occluded by debris or obstacles. Therefore, the study of pedestrian re-identification in occluded scenarios has important practical application value and deserves further study.

[0003] In the task of occluded person re-identification, the constructed person re-identification model usually includes an encoder and a decoder. The encoder is used to extract human features in the input image, and the decoder decodes the human features to obtain human part features, classifies the human part features or compares them with the features in the pedestrian database to obtain the pedestrian re-identification results. The effective fusion of different fine-grained features in the encoder has a significant impact on the performance of the pedestrian re-identification model. However, existing pedestrian re-identification methods usually ignore the impact of the fine-grained features of pedestrians in the encoder on the performance of pedestrian re-identification.

[0004] In addition, due to the limited size of the existing pedestrian re-identification dataset, a teacher-student decoder is introduced in the training process of the pedestrian re-identification model. Although the training is performed through the knowledge transfer method based on knowledge distillation, the accuracy of part feature matching can be improved. However, in the knowledge distillation process between the teacher decoder and the student decoder, it is difficult to achieve efficient knowledge transfer by simply using cosine similarity. The student decoder fails to effectively capture the knowledge of the teacher decoder, thus limiting the full performance of the pedestrian re-identification model. Summary of the invention

[0005] The present invention aims to at least solve the above-mentioned technical problems existing in the prior art and provide a pedestrian re-identification method and a training method for optimizing fine-grained feature fusion.

[0006] In order to achieve the above-mentioned object of the present invention, according to a first aspect of the present invention, the present invention provides a pedestrian re-identification method for optimizing fine-grained feature fusion, comprising:

[0007] Convert the image to be detected into a flattened sequence, wherein the class token is added to the flattened sequence;

[0008] Input the flattened sequence to the trained person re-identification model to obtain the pedestrian part features;

[0009] Re-identify the pedestrian's part features to obtain the pedestrian re-identification result;

[0010] Among them, the trained pedestrian re-identification model includes:

[0011] An adaptive multi-granularity feature fusion encoder is used to extract pedestrian features from a flattened sequence; the adaptive multi-granularity feature fusion encoder includes a stacked multi-layer Transformer encoding module and one or more multi-granularity feature fusion modules; wherein each multi-granularity feature fusion module is located between two adjacent layers of Transformer encoding modules, and is used to weightedly fuse the output features of all layers of Transformer encoding modules before the multi-granularity feature fusion module and the flattened sequence to obtain multi-granularity optimized features, and input the multi-granularity optimized features into the next layer of Transformer encoding module of the multi-granularity feature fusion module or output them as pedestrian features;

[0012] The decoder decodes the pedestrian features to obtain pedestrian part features, wherein the pedestrian part features include features of p parts of the pedestrian, and p is a positive integer.

[0013] Further preferably, each multi-granularity feature fusion module includes:

[0014] The text-visual feature interaction module inputs the query embedding and text-visual features learned in the person re-identification model training into the first multi-head attention layer for interactive calculation to obtain the first query feature;

[0015] The query image interaction module concatenates the class tokens in the output features of the Transformer encoding modules of all layers before the multi-granularity feature fusion module and the class tokens in the flattened sequence to obtain a concatenated token, and inputs the first query feature and the concatenated token into the second multi-head attention layer for interactive calculation to obtain a second query feature;

[0016] The feature weighting module uses the non-zero elements of the diagonal matrix of the second query feature as the fusion weights corresponding to the output features and flattened sequences of the Transformer encoding module before the multi-granularity feature fusion module in sequence, performs weighted processing on the output features and flattened sequences of all layers of Transformer encoding modules before the multi-granularity feature fusion module according to the fusion weights to obtain multi-granularity optimized features, and inputs the multi-granularity optimized features into the Transformer encoding module of the next layer of the multi-granularity feature fusion module or as pedestrian features.

[0017] Beneficial technical effects of the above technical solution:

[0018] 1. The adaptive multi-granularity feature fusion encoder uses multi-layer Transformer encoding modules set up in a stacked manner to extract features of different granularities respectively. In order to optimize the fusion of different granularity features output by the multi-layer Transformer encoding modules, a multi-granularity feature fusion module is set between two adjacent layers of Transformer encoding modules. The output features of the previous multi-layer Transformer encoding modules are weightedly fused, so that features at different levels are selectively added to the multi-granularity feature fusion module, and the fusion of each layer of features is gradually optimized from shallow to deep layers. It can capture the local details and global contours of pedestrians at the same time, improve the richness of feature representation, and thus improve model performance.

[0019] 2. The multi-granularity feature fusion module adaptively calculates the fusion weight of each layer of features. The text-visual feature interaction module and the query-image interaction module integrate the visual and text-visual features to help query embedding and learn character prototypes. The first multi-head attention layer and the second multi-head attention layer aggregate the image pedestrian information in the text-visual features and global features to obtain pedestrian prototypes of different granularities, thereby improving the modeling capability of pedestrian prototypes.

[0020] 3. It can better handle pedestrians at different distances and angles, enhance the robustness to granularity changes and the generalization ability of image enhancement features, help distinguish pedestrians with similar appearances, and ultimately improve the recognition accuracy of the pedestrian re-identification model, adapt to complex scenes, and perform better in crowded or partially occluded environments.

[0021] In order to achieve the above-mentioned object of the present invention, according to a second aspect of the present invention, the present invention provides a pedestrian re-identification model training method, comprising:

[0022] Obtain an image sample set, set a re-identification label and a human body part analysis label for each image sample, and divide a training set from the image sample set;

[0023] Constructing a re-identification training model, wherein the re-identification training model includes an encoder, a teacher decoder, a student decoder, and a mask generator;

[0024] The re-identification training model is trained using the training set until a training stop condition is reached to obtain a person re-identification model, wherein the person re-identification model includes an encoder and a student decoder connected in sequence;

[0025] The method of training the re-identification training model using the training set includes performing a second-stage training on the re-identification training model to obtain a pedestrian re-identification model. In each training of the second-stage training, the following steps are performed:

[0026] The image samples of the training set are converted into training flattened sequences, and the training flattened sequences are added with classtokens;

[0027] Input the training flattened sequence to the encoder to obtain pedestrian features;

[0028] The mask generator generates a pedestrian part mask according to pedestrian features, and the pedestrian part mask includes sub-masks of p parts of the pedestrian;

[0029] The teacher decoder decodes the feature area corresponding to the pedestrian part mask in the pedestrian feature to obtain the teacher pedestrian part feature, wherein the teacher pedestrian part feature includes the features of p parts of the pedestrian, where p is a positive integer;

[0030] The student decoder decodes the pedestrian features to obtain the student pedestrian part features, wherein the student pedestrian part features include the features of p parts of the pedestrian;

[0031] The total loss is calculated based on pedestrian features, pedestrian part masks, teacher pedestrian part features, student pedestrian part features, re-identification labels, and human body part parsing labels. The total loss includes pedestrian re-identification loss, part feature similarity loss, mask loss, and distillation loss.

[0032] The back-propagation algorithm is used to update the network parameters of the re-identification training model according to the total loss.

[0033] Further preferably, the re-identification training model further includes a distillation loss module for calculating the adaptive mask generation distillation loss, and the distillation loss module includes:

[0034] The first multi-head self-attention layer performs multi-head self-attention processing on the student pedestrian part features to obtain the student self-attention features;

[0035] An adaptive mask generation unit generates adaptive masks based on the student’s self-attention features;

[0036] A multiplication unit multiplies the adaptive mask with the three-dimensional student pedestrian part features to obtain the unmasked features of the pedestrian;

[0037] The convolution unit performs convolution processing on the unmasked features of pedestrians to obtain the features of the reconstructed student pedestrian parts;

[0038] The loss calculation unit calculates the adaptive mask generation distillation loss based on the reconstructed student pedestrian part features and the three-dimensional teacher pedestrian part features, and uses the adaptive mask generation distillation loss as the distillation loss.

[0039] Beneficial technical effects of the above technical solution:

[0040] 1. Construct a re-identification training model and use the teacher-student network framework to train the pedestrian re-identification model. A mask generator is set in the re-identification training model to improve the accuracy of the teacher decoder in obtaining the teacher pedestrian part features, so that the teacher decoder can guide the student decoder more accurately and effectively, thereby improving the accuracy of the student decoder output features.

[0041] 2. A distillation loss module is introduced to calculate the adaptive mask to generate the distillation loss. It can generate an adaptive mask, selectively block the student pedestrian part features output by the student decoder based on the adaptive mask, and use the convolution unit to obtain the reconstructed student pedestrian part features, so as to dynamically adjust the learning difficulty of the student model in the process of knowledge distillation and gradually master the complex features, which not only reduces the learning difficulty, but also ensures the quality of knowledge transfer and promotes the student decoder to absorb the teacher decoder knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of a pedestrian re-identification method for optimizing fine-grained feature fusion in a preferred embodiment of the present invention;

[0043] Figure 2 is a structural schematic diagram of a pedestrian re-identification model in a preferred embodiment of the present invention;

[0044] Figure 3 It is a structural schematic diagram of a multi-granularity feature fusion module in a preferred embodiment of the present invention;

[0045] Figure 4 Schematic diagram of the network structure of an adaptive multi-granularity feature fusion encoder of a pedestrian re-identification model in a preferred embodiment of the present invention;

[0046] Figure 5 It is a flowchart of a pedestrian re-identification model training method in a preferred embodiment of the present invention;

[0047] Figure 6 It is a schematic diagram of each training process of the re-identification training model in a preferred embodiment of the present invention;

[0048] Figure 7 is a network structure diagram of a re-identification training model in a preferred embodiment of the present invention;

[0049] Figure 8 is a schematic structural diagram of a distillation loss module in a preferred embodiment of the present invention;

[0050] Fig. 9 is a network structure diagram of a distillation loss module in a preferred embodiment of the present invention;

[0051] Fig.10It is a schematic structural diagram of an electronic device in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0052] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0053] The execution subject of the pedestrian re-identification method or pedestrian re-identification model training method for optimizing fine-grained feature fusion provided by the present invention includes but is not limited to a server, a terminal, etc., which can be configured to execute at least one of the electronic devices of the method provided in the embodiment of the present application.

[0054] The present invention provides a pedestrian re-identification method for optimizing fine-grained feature fusion. In a preferred embodiment, Figure 1 As shown, including:

[0055] Step A1: convert the image to be detected into a flattened sequence, and add the class token to the flattened sequence.

[0056] In this embodiment, the image to be detected is a pedestrian image captured by a camera in a scene where pedestrian re-identification is required. There may or may not be pedestrians in the pedestrian image. In order to adapt to the sequence processing mechanism of Transformer, the image to be detected needs to be converted into a flattened sequence. Figure 4 As shown, a flattened sequence conversion unit may be provided to convert the image to be detected into a flattened sequence. Preferably, the flattened sequence conversion unit performs:

[0057] Step 1: Divide the image to be detected into multiple image blocks, set to N image blocks, where N is a positive integer, and map the image blocks to a preset multidimensional feature space to obtain a mapping token sequence.

[0058] Assume the image to be detected is Where H, W, and C represent the height, width, and number of channels of the detection image, respectively. Use a sliding window to divide the input image into N fixed-size image blocks. The N image blocks do not overlap, and the number of blocks N is calculated by the following formula:

[0059]

[0060] Among them, S represents the sliding step size, P is the image block size, is a floor rounding function. The image block is mapped to the D-dimensional feature space through linear projection, and one image block obtains a mapping token, where token represents a token.

[0061] Step 2: Add the class token (x class ), position encoding, and embedding the camera view to obtain a flattened sequence. Class token represents the category token, which is used to aggregate global features. The class token added in actual application is learned after the pedestrian re-identification model training is completed. In the pedestrian re-identification model training, classtoken is continuously learned and updated. By introducing position encoding to maintain the spatial information of the detection image, integrating camera view embedding to enhance the view invariance, so the final two-dimensional flattened sequence can be expressed as

[0062] Step A2: Input the flattened sequence to the trained person re-identification model to obtain the pedestrian part features. Figure 2 and Figure 4 As shown in the figure, the trained pedestrian re-identification model includes:

[0063] (1) An adaptive multi-granularity feature fusion encoder for extracting pedestrian features from the flattened sequence; the adaptive multi-granularity feature fusion encoder includes a stacked multi-layer Transformer encoding module (i.e. Figure 4 TransformerBlock in the figure), and one or more multi-granularity feature fusion modules (abbreviated as AMFFM); wherein each multi-granularity feature fusion module AMFFM is located between two adjacent layers of Transformer encoding modules, and is used to weightedly fuse the output features of all layers of Transformer encoding modules before the multi-granularity feature fusion module and the flattened sequence to obtain multi-granularity optimized features, and input the multi-granularity optimized features into the Transformer encoding module of the next layer of the multi-granularity feature fusion module or output them as pedestrian features.

[0064] Reference Figure 4As shown, assume that the adaptive multi-granularity feature fusion encoder is stacked with L layers of Transformer encoding modules, and the Transformer encoding module index is k, k∈[1,L]. The network structure of the Transformer encoding module can refer to the encoding block (Encoding Block) of the existing Vision Transformer network, which will not be repeated here. In the adaptive multi-granularity feature fusion encoder, only one multi-granularity feature fusion module is set between two adjacent layers of Transformer encoding modules. In an example, a multi-granularity feature fusion module AMFFM is connected after the Transformer encoding modules of layers 1, 2, 4, 6, 8, and 10 respectively. It is assumed that a multi-granularity feature fusion module AMFFM is set after the Transformer encoding module of the kth layer, which is used to perform weighted fusion of the output features of the Transformer encoding modules of layers 1 to k and the flattened sequence.

[0065] (2) Decoder: decodes the pedestrian features to obtain pedestrian part features, where the pedestrian part features include features of p parts of the pedestrian, where p is a positive integer. The decoder structure can refer to the network structure of the existing Vision Transformer decoder, which will not be described here.

[0066] Step A3: re-identify the pedestrian's part features to obtain a pedestrian re-identification result.

[0067] In this embodiment, the pedestrian re-identification result includes the probability that the pedestrian in the image to be detected belongs to each pedestrian in the preset pedestrian library and the probability that there is no pedestrian in the image to be detected. The preset pedestrian library can be a human body part feature library of the pedestrian to be tracked or the pedestrian previously identified.

[0068] In one example, a classifier composed of more than one cascaded fully connected layers is used to process the pedestrian part features to obtain the pedestrian re-identification result. In another example, step A3 may include: calculating the similarity between the obtained pedestrian part features and each human body part feature in the preset pedestrian library, when all similarities are greater than the preset similarity threshold, the pedestrian corresponding to the human body similarity feature with the greatest similarity to its human body part feature in the preset pedestrian library is used as the re-identification result of the image to be detected, and when all similarities are less than the preset similarity threshold, it is considered that there is no pedestrian in the image to be detected. Alternatively, the distance (such as Euclidean distance) between the obtained pedestrian part features and each human body part feature in the preset pedestrian library is calculated, and the pedestrians are output in ascending order according to the distance.

[0069] In a preferred embodiment, when the existing VisionTransformer using stacked Transformer encoding modules extracts pedestrian features, the shallow Transformer Block mainly captures local features (such as textures and edges), and the deep Transformer Block focuses on global semantic features (such as the entire pedestrian and the scene). The output of the k∈[1,L]th layer Transformer Block can be regarded as a weighted fusion of features from the 0th layer (input sequence) to the kth layer, where the weight of each layer of features is 1. This fixed-weight fusion method cannot dynamically adapt to the needs of pedestrian feature extraction in complex scenes. Therefore, the present application inserts one or more multi-granularity feature fusion modules between stacked multi-layer Transformer encoding modules to achieve dynamic calculation and adaptive fusion of feature weights. Preferably, as Figure 3 and Figure 4 As shown, each multi-granularity feature fusion module includes:

[0070] (1) The text-visual feature interaction module inputs the query embedding learned in the person re-identification model training (learned in the second stage of training) and the learned text-visual features (learned in the first stage of training) into the first multi-head attention layer for interactive calculation to obtain the first query feature.

[0071] In the person re-identification model training: for each layer of Transformer encoding module (Transformer block), a set of learnable query embeddings (i.e., query embeddings) is defined. For the k-th layer Transformer block, its query embedding is expressed as In order to obtain the visual features of the text, the pre-trained CLIP model (Contrastive Language-Image Pre-Training, a multimodal pre-trained neural network) is introduced in the first stage of training to generate the visual features of the text based on the text description. The two-stage training strategy of CLIPReID can be used as a reference. The input text description of the CLIP model is “A photo of a[X]1[X]2…[X] M person", each of which is a learnable text token, M represents the number of text tokens, and m represents the text token index. Freeze all parameters except text tokens in the person re-identification model, and input the text description into the text encoder of the frozen CLIP model to extract the initial text visual features. By calculating the alignment loss between the text visual features and the pedestrian features extracted by the adaptive multi-granularity feature fusion encoder, the text token is gradient updated to enhance the cross-modal adaptation ability of the text visual features. After the training is completed, freeze the CLIP model and text tokens, save the text visual features output by the CLIP model text encoder, and the CLIP model will no longer participate in the subsequent second-stage training. In the subsequent second-stage training, the text visual feature interaction module directly calls the saved text visual features to calculate the first query feature. The first query feature representation E' obtained by the text visual feature interaction module of the multi-granularity feature fusion module connected after the k-th layer Transformer Block k for:

[0072] E' k =MHA1(E k ,T,E k )

[0073] Among them, MHA1 represents the first multi-head attention layer, T represents the saved learned text visual features, so that the information of people in the text-visual features is gathered to This helps the query learn character prototypes from textual visual features with strong generalization ability, thereby improving the understanding of the basic concept of people.

[0074] (2) Query the image interaction module, concatenate the class tokens in the output features of all layers of the Transformer encoding module before the multi-granularity feature fusion module and the class tokens in the flattened sequence to obtain a concatenated token, input the first query feature and the concatenated token into the second multi-head attention layer for interactive calculation, and obtain the second query feature.

[0075] In order to obtain image features of different granularities in ViTs, for the multi-granularity feature fusion module connected after the k-th layer Transformer Block, the class tokens embedded in the output features from the 0th layer (i.e., the flattened sequence of the input) to the k-th layer Transformer encoding module are concatenated, where the i-th class token is represented as g i , The obtained concatenated token is represented by G k =[g0,g1,…,g k ],in E'k and G k Input into the second multi-head attention layer to realize the query-image interaction, and obtain the second query feature representation as:

[0076] E” k =MHA2(G k ,E' k ,G k )

[0077] in, Through two attention layers, the pedestrian information in the text features and global features is aggregated to obtain pedestrian prototypes of different granularities. MHA2 represents the second multi-head attention layer.

[0078] (3) A feature weighting module, which uses the non-zero elements of the diagonal matrix of the second query feature as the fusion weights corresponding to the output features and flattened sequences of all layers of Transformer encoding modules before the multi-granularity feature fusion module in turn, performs weighted processing on the output features and flattened sequences of all layers of Transformer encoding modules before the multi-granularity feature fusion module according to the fusion weights to obtain multi-granularity optimized features, and inputs the multi-granularity optimized features into the Transformer encoding module of the next layer of the multi-granularity feature fusion module or as pedestrian features.

[0079] In this embodiment, the diagonal matrix of the second query feature is used as the fusion weight of each layer of features. Each layer of features is multiplied by the corresponding fusion weight, and a gating mechanism is introduced. By adaptively adjusting the fusion weights of features of different granularities, the generalization ability of the output features of Vision Transformer is enhanced, and the performance of the model in complex scenes is improved.

[0080] In this embodiment, preferably, the feature weighting module performs weighted processing on the output features and flattened sequences of all layers of Transformer encoding modules before the multi-granularity feature fusion module according to the fusion weights to obtain multi-granularity optimization features, including:

[0081] Step A31, multiplying the flattened sequence by its corresponding fusion weight to obtain the intermediate feature corresponding to the flattened sequence;

[0082] Step A32, multiplying the output features of each layer of Transformer encoding module before the multi-granularity feature fusion module with its corresponding fusion weight to obtain the intermediate features corresponding to each layer of Transformer encoding module;

[0083] Step A33, accumulating the intermediate features corresponding to the Transformer encoding modules of all layers before the multi-granularity feature fusion module and the intermediate features corresponding to the flattened sequence to obtain the accumulated features;

[0084] Step A34, the concatenated token is processed by a first activation function to obtain a superposition weight vector; the first activation function is preferably but not limited to a sigmoid activation function. The superposition weight vector includes a first superposition weight and a second superposition weight, the second superposition weight is assigned to the accumulated feature, and the first superposition weight is assigned to the output feature of the previous layer Transformer encoding module of the multi-granularity feature fusion module.

[0085] Step A35, using the superimposed weight vector, weightedly sums the accumulated features and the output features of the previous layer Transformer encoding module of the multi-granularity feature fusion module to obtain the multi-granularity optimization features.

[0086] In one example, for the multi-granularity feature fusion module connected after the k-th layer Transformer Block, the multi-granularity optimized feature representation x' k for:

[0087]

[0088] α k =σ(g k )

[0089] Among them, diag() means converting the vector into a diagonal matrix, diag((E” k ) i ) means E” k The non-zero elements in the i-th row of the diagonal matrix of α k represents the second superposition weight, (1-α k ) represents the first superposition weight, x' k It represents the optimized output of the multi-granularity feature fusion module AMFFM connected after the k-th layer Transformer Block, which will be used as the input of the next layer Transformer Block or as the pedestrian feature output of the encoder. i represents the output feature of the i-th layer TransformerBlock, σ(·) is the first activation function, which is the sigmoid activation function, and g k Indicates concatenated tokens.

[0090] In this embodiment, the multi-granularity feature fusion module is embedded into the stacked Transformer Block to achieve adaptive fusion of multi-granularity features. By selectively embedding the multi-granularity feature fusion module AMFFM in different levels of the Vision Transformer, an adaptive multi-granularity feature fusion encoder is constructed, which can optimize the fusion of multi-granularity features from the bottom up, significantly improving the generalization ability of features and the robustness to pedestrian occlusion scenes. It is empirically selected to embed the multi-granularity feature fusion module AMFFM in the {1, 2, 4, 6, 8, 10}th layer Transformer Block of the adaptive multi-granularity feature fusion encoder. The pedestrian features extracted by the adaptive multi-granularity feature fusion encoder can be expressed as F=[F g ,F local ],in The class token is N mapping tokens form a sequence

[0091] In a preferred embodiment, referring to Figure 7 The student decoder structure includes:

[0092] The self-attention module uses the self-attention mechanism to process the pedestrian features output by the adaptive multi-granularity feature fusion encoder to obtain self-attention features;

[0093] The cross-attention module uses the cross-attention mechanism to process the pedestrian features output by the self-attention features and the adaptive multi-granularity feature fusion encoder to obtain pedestrian part features. In the cross-attention mechanism, the pedestrian features are used as the V and K matrices of attention, and the self-attention features are used as the Q matrix of attention.

[0094] In this implementation, the training process of the person re-identification model training includes:

[0095] (1) Construct an image sample set and set a re-identification label for each image sample; convert each image sample into a training flattened sequence; the image samples include pedestrians, and existing datasets such as Occluded Duke, Market1501, Duke-MTMC, OCC-ReID, and P-Duke-MTMC can be used.

[0096] (2) Construct an adaptive multi-granularity feature fusion encoder, decoder, classifier, and CLIP model, and initialize network parameters. Randomly initialize query embedding, classtoken, and text tokens in text descriptions;

[0097] (3) First stage training: Loop through the following steps until the alignment loss satisfies the alignment loss condition (alignment loss or less than the alignment loss threshold):

[0098] Step a1, input the text description with text TOKEN into the text encoder of the CLIP model to obtain text visual features;

[0099] Step a2, input the training flattened sequence corresponding to the sample image to the encoder to obtain pedestrian features;

[0100] Step a3, calculate the alignment loss through text visual features and pedestrian features; the alignment loss calculation method of CLIPReID can be referred to, which will not be repeated here.

[0101] Step a4: Update the gradient of the text token according to the alignment loss.

[0102] The second stage of training includes: inputting the text description embedded with the text tokens obtained after the first stage of training into the text encoder of the CLIP model to obtain the learned text visual features and freeze the text visual features; defining a set of learnable query embeddings for each layer of the Transformer encoding module (Transformer block); forming a training model with an adaptive multi-granularity feature fusion encoder, decoder, and classifier, dividing the image sample set into a training set, a test set, and a validation set, training the training model with the training set, and testing and validating the trained model with the test set and validation set, respectively. After the test and validation pass, the training model is used as a pedestrian re-identification model, or the adaptive multi-granularity feature fusion encoder and decoder in the training model are connected as a pedestrian re-identification model.

[0103] During each training step of the model:

[0104] The training flattened sequence corresponding to the training image is input into the adaptive multi-granularity feature fusion encoder to obtain pedestrian features, wherein the text visual feature interaction module of the adaptive multi-granularity feature fusion encoder obtains the first query feature according to the text visual features frozen in the first stage of training.

[0105] The decoder decodes the pedestrian features to obtain the pedestrian part features;

[0106] The classifier obtains the pedestrian re-identification result according to the pedestrian part features, calculates the re-identification loss according to the pedestrian re-identification result and the re-identification label of the training sample image, and uses the back propagation algorithm to update the network parameters, query embedding and class token of the training model according to the re-identification loss. The calculation process of the re-identification loss is existing, such as the re-identification loss calculation method disclosed in the Chinese patent publication number CN118690822A, which will not be repeated here.

[0107] The present invention also discloses a pedestrian re-identification model training method. In a preferred embodiment, referring to Figure 5 and Figure 6 ,include:

[0108] Step B1, obtain an image sample set, set a re-identification label and a human body part parsing label for each image sample, and divide the image sample set into a training set, a test set, and a validation set.

[0109] In this embodiment, the source of the image sample set can refer to the above training method, which will not be repeated here. Preferably, the image sample is processed by a human parsing model (Human Parsing) to obtain a human parsing label, referring to Figure 7 shown.

[0110] Step B2: construct a re-identification training model, which includes an encoder, a teacher decoder, a student decoder, and a mask generator. The teacher decoder and the mask generator are auxiliary models for the training of the pedestrian re-identification model.

[0111] In this embodiment, the encoder is preferably but not limited to a single-layer Transformer block or a stacked multi-layer Transformer block in VisionTransformer.

[0112] In this embodiment, the network structure of the teacher decoder and the student decoder refers to the decoder structure in the above-mentioned pedestrian re-identification method for optimizing fine-grained feature fusion. Specifically, the student decoder includes a self-attention module 1 and a cross-attention module connected in sequence. The teacher decoder includes a self-attention module 2 and a mask cross-attention module connected in sequence, and the mask cross-attention module includes an extraction module and a cross-attention module. The extraction module is used to extract the feature area corresponding to the pedestrian part mask in the pedestrian feature, and the cross-attention module performs a cross-attention mechanism on the extracted feature area to obtain the teacher's body part feature.

[0113] In this embodiment, the mask generator includes a fully connected layer and a softmax activation function layer connected in sequence.

[0114] Step B3, using the training set to train the re-identification training model until the training stop condition is reached, and a pedestrian re-identification model is obtained, wherein the pedestrian re-identification model includes an encoder and a student decoder connected in sequence. After the training is completed, the trained re-identification training model is tested and verified by the test set and the verification set respectively, and the pedestrian re-identification model is obtained after the test and verification pass. The training stop condition is not limited to that the number of iterative training times reaches the maximum number of prediction times or the total loss is less than the loss threshold.

[0115] Among them, using the training set to train the re-identification training model includes performing a second-stage training on the re-identification training model to obtain a pedestrian re-identification model. In each training of the second-stage training, Figure 6 As shown, execute:

[0116] Step B31, converting the image samples of the training set into a training flattened sequence, and adding a class token to the training flattened sequence; the class token is learnable, and the added class token is learned in the last training.

[0117] Step B31, input the training flattened sequence to the encoder to obtain pedestrian features.

[0118] Step B31: The mask generator generates a pedestrian part mask according to pedestrian features, and the pedestrian part mask includes sub-masks of p parts of the pedestrian.

[0119] Step B31, the teacher decoder decodes the feature area in the pedestrian feature corresponding to the pedestrian part mask to obtain the teacher pedestrian part feature, wherein the teacher pedestrian part feature includes the features of p parts of the pedestrian, and p is a positive integer.

[0120] Step B31, the student decoder decodes the pedestrian features to obtain the student pedestrian part features, wherein the student pedestrian part features include features of p parts of the pedestrian.

[0121] Step B31, calculate the total loss based on pedestrian features, pedestrian part masks, teacher pedestrian part features, student pedestrian part features, re-identification labels and human body part analysis labels. The total loss includes the pedestrian re-identification loss L reid , Part feature similarity loss L div , mask loss L m and distillation loss L AMGD .

[0122] The total loss is calculated as:

[0123] L=L reid +L m +L AMGD +L div

[0124] The calculation methods of the above three losses can refer to the existing technology, such as the calculation method disclosed in the paper entitled "PART REPRESENTATION LEARNING WITH TEACHER-STUDENT DECODER FOR OCCLUDED PERSON RE-IDENTIFICATION" published by Shang Gao et al. in 2023.AMGD It can be the distillation loss in this paper. L ce is the cross entropy loss with the BNNeck trick. L tri is the triplet loss. and It is generated by concatenating body part features along the channel dimension. is the average triplet loss of body parts. L m Used for training to generate masks, L AMGD This may cause the decoder to extract the same features, so the diversity loss L is used on the part features. div You can refer to the above-mentioned papers for details, which will not be elaborated here.

[0125] Step B31, using the back propagation algorithm to update the network parameters of the re-identification training model according to the total loss.

[0126] In a preferred embodiment, the re-identification training model further includes a distillation loss module for calculating the distillation loss for adaptive mask generation, referring to Figure 8 and Fig. 9 , the distillation loss module includes:

[0127] (1) In the first multi-head self-attention layer, multi-head self-attention processing is performed on the student pedestrian part features to obtain the student self-attention features. The student pedestrian part features are expressed as The teacher pedestrian part feature is expressed as p represents the number of human parts (can be set as needed), and the student pedestrian part features and the teacher pedestrian part features are reshaped into 3D forms, respectively, and are represented as the three-dimensional student pedestrian part features S and the three-dimensional teacher pedestrian part features Extract all learnable class tokens from S to form a token vector V. Input the token vector V into the first multi-head self-attention layer to obtain the student self-attention feature R':

[0128] R'=MHSA(V)

[0129] MHSA represents the first multi-head self-attention layer, Reshape R' into 3D to obtain 3D student self-attention features

[0130] (2) Adaptive mask generation unit generates an adaptive mask based on the three-dimensional student self-attention features.

[0131] (3) A multiplication unit, which multiplies the adaptive mask mask with the three-dimensional student pedestrian part feature S to obtain the pedestrian unmasked feature S·mask;

[0132] (4) Convolution unit, which performs convolution processing on the unmasked features S mask of the pedestrian to obtain the reconstructed student pedestrian part features S':

[0133] It means that the convolution unit includes two 3×3 convolution layers: W l1 and W l2 , and a ReLU activation layer, which can be expressed as: F stands for Features to process.

[0134] (5) Loss calculation unit, which calculates the adaptive mask generation distillation loss based on the reconstructed student pedestrian part features and the three-dimensional teacher pedestrian part features, and uses the adaptive mask generation distillation loss as the distillation loss Ioss AMGD The mean square error can be used to calculate the distillation loss loss AMGD :

[0135] loss AMGD =MSE(Sa,Te)×a amgd ;

[0136] Among them, a amgd represents the first hyperparameter; MSE(·) represents the mean square error. Through adaptive masking, the student model can learn the knowledge of important feature areas first and gradually master complex features, thereby reducing the learning difficulty while ensuring the quality of knowledge transfer.

[0137] In a preferred embodiment, the adaptive mask generation unit performs:

[0138] Step b1, generate a mask template that is the same as the three-dimensional student self-attention feature.

[0139] Step b2, traverse the feature points (height-width plane feature points) of the three-dimensional student self-attention feature and execute: calculate the difference between the feature value of the current traversed feature point and the feature threshold. If the difference is greater than the difference threshold, set the feature value of the feature point corresponding to the position of the current traversed feature point in the mask template to 1; if the difference is less than or equal to the difference threshold, set the feature value of the feature point corresponding to the position of the current traversed feature point in the mask template to 0; the difference threshold is preferably but not limited to 0. The height-width plane (H×W) position coordinates of the feature points of the three-dimensional student self-attention feature are expressed as [u, v], and the feature value mask[u, v] of the feature point at position [u, v] in the mask template is obtained according to the following formula:

[0140] mask[u,v]=H(R[u,v]−λ amgd )

[0141] Where R[u, v] represents the feature value of the feature point at the position [u, v] of the three-dimensional student self-attention feature; λ amgd represents the occlusion ratio, which is a hyperparameter; H(z) represents the mask generation function, and z represents the input feature of the mask generation function:

[0142]

[0143] Step b3, after traversing all feature points of the three-dimensional student self-attention feature, the mask template with updated feature values ​​of all feature points is used as an adaptive mask.

[0144] In a preferred embodiment, the re-identification training model further includes a CLIP model, and a text encoder of the CLIP model is used to convert the text description into text visual features, and a learnable text token is embedded in the text description;

[0145] The encoder is an adaptive multi-granularity feature fusion encoder, such as Figure 4 and 3 As shown, the adaptive multi-granularity feature fusion encoder includes a stacked multi-layer Transformer encoding module and more than one multi-granularity feature fusion module; wherein each multi-granularity feature fusion module is located between two adjacent layers of Transformer encoding modules, and is used to weightedly fuse the output features of all layers of Transformer encoding modules before the multi-granularity feature fusion module and the flattened sequence to obtain multi-granularity optimization features, and input the multi-granularity optimization features into the latter layer of Transformer encoding module of the multi-granularity feature fusion module or output them as pedestrian features;

[0146] Each multi-granularity feature fusion module includes:

[0147] The text-visual feature interaction module inputs the query embedding (learned in the second stage of training) and the text-visual features (learned in the first stage of training) learned in the person re-identification model training into the first multi-head attention layer for interactive calculation to obtain the first query feature;

[0148] The query image interaction module concatenates the class tokens in the output features of the Transformer encoding modules of all layers before the multi-granularity feature fusion module and the class tokens in the flattened sequence to obtain a concatenated token, and inputs the first query feature and the concatenated token into the second multi-head attention layer for interactive calculation to obtain a second query feature;

[0149] The feature weighting module uses the non-zero elements of the diagonal matrix of the second query feature as the fusion weights corresponding to the output features and flattened sequences of all layers of Transformer encoding modules before the multi-granularity feature fusion module in sequence, performs weighted processing on the output features and flattened sequences of all layers of Transformer encoding modules before the multi-granularity feature fusion module according to the fusion weights to obtain multi-granularity optimized features, and inputs the multi-granularity optimized features into the Transformer encoding module of the next layer of the multi-granularity feature fusion module or as pedestrian features.

[0150] In this embodiment, the specific structure and working principle of the CLIP model and the adaptive multi-granularity feature fusion encoder have been described in the above-mentioned disclosed method for optimizing fine-grained feature fusion for pedestrian re-identification, and will not be repeated here.

[0151] In this embodiment, after the CLIP model and the encoder are set as an adaptive multi-granularity feature fusion encoder, the above-mentioned re-identification training model using the training set is to first perform the first stage training and then perform the second stage training. The first stage training is to loop the following steps until the alignment loss meets the alignment loss condition (the alignment loss is less than the alignment loss threshold):

[0152] Step C1, input the text description with text TOKEN into the text encoder of the CLIP model to obtain text visual features;

[0153] Step C2, input the training flattened sequence corresponding to the sample image to the encoder to obtain pedestrian features;

[0154] Step C3, calculate the alignment loss through text visual features and pedestrian features; the alignment loss calculation method of CLIPReID can be referred to and will not be repeated here.

[0155] Step C4, update the gradient of the text token according to the alignment loss;

[0156] The second stage of training includes: inputting the text description embedded with the text tokens obtained after the first stage of training into the text encoder of the CLIP model, obtaining the learned text visual features, and freezing the text visual features; Figure 7 As shown in the figure, the adaptive multi-granularity feature fusion encoder, teacher decoder, student decoder, mask generator, and distillation loss module together constitute a new re-identification training model. The new re-identification training model is trained using the training set, and the trained new re-identification training model is tested and verified using the test set and verification set, respectively. After the test and verification are passed, the adaptive multi-granularity feature fusion encoder and student decoder in the new re-identification training model are connected in sequence to form a pedestrian re-identification model.

[0157] In each training run of a new re-ID training model, execute:

[0158] Step D1: define a set of learnable query embeddings for each layer of Transformer encoding module (Transformer block); add the class token learned in the last training to the training flattened sequence.

[0159] Step D2, input the training flattened sequence corresponding to the training image into the adaptive multi-granularity feature fusion encoder to obtain pedestrian features, wherein the text visual feature interaction module of the adaptive multi-granularity feature fusion encoder obtains the first query feature according to the text visual features frozen in the first stage of training.

[0160] In step D2, a mask generator generates a pedestrian part mask according to pedestrian features, and the pedestrian part mask includes sub-masks of p parts of the pedestrian.

[0161] Step D3, the teacher decoder decodes the feature area in the pedestrian feature corresponding to the pedestrian part mask to obtain the teacher pedestrian part feature, wherein the teacher pedestrian part feature includes the features of p parts of the pedestrian, and p is a positive integer.

[0162] Step D4, the student decoder decodes the pedestrian features to obtain student pedestrian part features, wherein the student pedestrian part features include features of p parts of the pedestrian.

[0163] Step D5: The distillation loss module calculates the distillation loss L according to the student pedestrian part features and the teacher pedestrian part features. AMGD .

[0164] Step D6, calculate the person re-identification loss L according to the pedestrian features, pedestrian part masks, teacher pedestrian part features, student pedestrian part features, re-identification labels and human body part parsing labels reid , Part feature similarity loss L div and distillation loss L AMGD . Obtain the total loss, including the distillation loss L AMGD Person re-identification loss L reid , Part feature similarity loss L div and distillation loss L AMGD . The back-propagation algorithm is used to update the network parameters, query embedding and classtoken of the training model according to the total loss.

[0165] Due to the limited size of existing person re-identification datasets, it is difficult to achieve accurate human body part feature extraction by relying solely on external models (i.e., encoders and decoders). Therefore, the above-mentioned person re-identification model training method provided by the present invention introduces a teacher-student decoder, which obtains pedestrian part prototypes by combining a learnable human body part mask with an external model, thereby achieving accurate part feature matching. However, the teacher-student knowledge distillation method based on cosine similarity is difficult to achieve effective knowledge transfer. To this end, an Adaptive MaskGenerative Distillation Loss (AMGDLoss), i.e., a distillation loss module, is proposed in the person re-identification model training method of the present application, which aims to optimize the knowledge distillation of the teacher-student network framework. The core of the method is to generate an adaptive mask, selectively block the prediction results of the student network, and use convolution operations to restore the blocked prediction information. The learning difficulty of the student model in the knowledge distillation process is dynamically adjusted, so that the teacher network can guide the student network more accurately and effectively. This method guides the student model to prioritize the knowledge of key features and gradually master complex features, which not only reduces the learning difficulty, but also ensures the quality of knowledge transfer.

[0166] In an example of the person re-identification model training method provided by the present invention, the adaptive multi-granularity feature fusion encoder includes 12 transformer encoding module layers with a hidden size of 768 dimensions. In order to match the output of the adaptive multi-granularity feature fusion encoder, the 512-dimensional text visual features extracted by the CLIP text encoder are expanded to 768 dimensions through a linear layer.

[0167] In the above example, in the first stage of training, the Adam optimizer is used and the learning rate is initialized to 3.5×10 -4 , and decayed by cosine scheduling. In this stage, the batch size is set to 64, the images are resized to 256×128, and are enhanced by random erasing, padding, and random cropping. Only learnable text tokens [X]1[X]2[X]3...[X] are optimized M The model was trained for 120 epochs with a linear layer that expanded the dimension of the text visual features from 512 to 768. In the second phase of training, the SGD optimizer was used to optimize the image encoder. All images were resized to 256×128 and augmented by random erasing, padding, and random cropping. The batch size was set to 64, with 4 images per ID. The momentum was 0.9 and the weight decay was 1e. -4 The learning rate is initialized to 0.004 and cosine learning rate decay is used. The model is trained for 120 epochs. The number of body parts p is empirically set to 8. The same human body parsing model as in BpbReid is used.

[0168] The present invention also discloses a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned pedestrian re-identification method for optimizing fine-grained feature fusion or a pedestrian re-identification model training method provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0169] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory connected to the at least one processor in communication; wherein:

[0170] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a pedestrian re-identification method for optimizing fine-grained feature fusion or a pedestrian re-identification model training method provided by the present invention.

[0171] like Fig.10 , is a schematic diagram of the structure of an electronic device for a pedestrian re-identification method for optimizing fine-grained feature fusion or a pedestrian re-identification model training method provided by an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a pedestrian re-identification method for optimizing fine-grained feature fusion or a pedestrian re-identification model training method program.

[0172] Among them, the processor 10 can be composed of an integrated circuit in some embodiments, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect the various components of the entire electronic device, and executes or executes the program or module stored in the memory 11 (for example, executing a pedestrian re-identification method for optimizing fine-grained feature fusion or a pedestrian re-identification model training method, etc.), and calls the data stored in the memory 11 to execute various functions of the electronic device and process data.

[0173] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as a pedestrian re-identification method for optimizing fine-grained feature fusion or a pedestrian re-identification model training method program code, etc., but can also be used to temporarily store data that has been output or is to be output.

[0174] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0175] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0176] Fig.10 Only an electronic device with components is shown, and those skilled in the art will understand that Fig.10The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0177] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

Claims

1. A pedestrian re-identification method for optimizing fine-grained feature fusion, characterized in that: include: Convert the image to be detected into a flattened sequence, wherein the class token is added to the flattened sequence; Input the flattened sequence to the trained person re-identification model to obtain the pedestrian part features; Re-identify the pedestrian's part features to obtain the pedestrian re-identification result; Among them, the trained pedestrian re-identification model includes: An adaptive multi-granularity feature fusion encoder is used to extract pedestrian features from a flattened sequence; the adaptive multi-granularity feature fusion encoder includes a stacked multi-layer Transformer encoding module and one or more multi-granularity feature fusion modules; wherein each multi-granularity feature fusion module is located between two adjacent layers of Transformer encoding modules, and is used to weightedly fuse the output features of all layers of Transformer encoding modules before the multi-granularity feature fusion module and the flattened sequence to obtain multi-granularity optimized features, and input the multi-granularity optimized features into the next layer of Transformer encoding module of the multi-granularity feature fusion module or output them as pedestrian features; The decoder decodes the pedestrian features to obtain pedestrian part features, wherein the pedestrian part features include features of p parts of the pedestrian, and p is a positive integer.

2. The pedestrian re-identification method for optimizing fine-grained feature fusion according to claim 1, characterized in that: The step of converting the image to be detected into a flattened sequence includes: Divide the image to be detected into multiple image blocks, and map the image blocks to a preset multi-dimensional feature space to obtain a mapping token sequence; Add class token, position encoding, and embedded camera view to the mapping token sequence to obtain a flattened sequence.

3. The pedestrian re-identification method for optimizing fine-grained feature fusion according to claim 1 or 2, characterized in that: Each multi-granularity feature fusion module includes: The text-visual feature interaction module inputs the query embedding and text-visual features learned in the person re-identification model training into the first multi-head attention layer for interactive calculation to obtain the first query feature; The query image interaction module concatenates the class tokens in the output features of the Transformer encoding modules of all layers before the multi-granularity feature fusion module and the class tokens in the flattened sequence to obtain a concatenated token, and inputs the first query feature and the concatenated token into the second multi-head attention layer for interactive calculation to obtain a second query feature; The feature weighting module uses the non-zero elements of the diagonal matrix of the second query feature as the fusion weights corresponding to the output features and flattened sequences of all layers of Transformer encoding modules before the multi-granularity feature fusion module in sequence, performs weighted processing on the output features and flattened sequences of all layers of Transformer encoding modules before the multi-granularity feature fusion module according to the fusion weights to obtain multi-granularity optimized features, and inputs the multi-granularity optimized features into the Transformer encoding module of the next layer of the multi-granularity feature fusion module or as pedestrian features.

4. The pedestrian re-identification method for optimizing fine-grained feature fusion according to claim 3, characterized in that: The feature weighting module performs weighted processing on the output features and flattened sequences of all layers of Transformer encoding modules before the multi-granularity feature fusion module according to the fusion weights to obtain multi-granularity optimization features, including: Multiply the flattened sequence by the corresponding fusion weight to obtain the intermediate features corresponding to the flattened sequence; The output features of each layer of Transformer encoding module before the multi-granularity feature fusion module are multiplied by the corresponding fusion weight to obtain the intermediate features corresponding to each layer of Transformer encoding module; Accumulate the intermediate features corresponding to the Transformer encoding modules of all layers before the multi-granularity feature fusion module and the intermediate features corresponding to the flattened sequence to obtain the accumulated features; Perform the first activation function on the concatenated token to obtain a superposition weight vector; The multi-granularity optimization features are obtained by weighted summing the accumulated features and the output features of the previous layer Transformer encoding module of the multi-granularity feature fusion module using the superimposed weight vector.

5. The pedestrian re-identification method for optimizing fine-grained feature fusion according to claim 1, 2 or 4, characterized in that: The decoder comprises: The self-attention module uses the self-attention mechanism to process the pedestrian features output by the adaptive multi-granularity feature fusion encoder to obtain self-attention features; The cross-attention module uses the cross-attention mechanism to process the pedestrian features output by the self-attention features and the adaptive multi-granularity feature fusion encoder to obtain the pedestrian part features.

6. A pedestrian re-identification model training method, characterized in that: include: Obtain an image sample set, set a re-identification label and a human body part analysis label for each image sample, and divide a training set from the image sample set; Constructing a re-identification training model, wherein the re-identification training model includes an encoder, a teacher decoder, a student decoder, and a mask generator; The re-identification training model is trained using the training set until a training stop condition is reached to obtain a person re-identification model, wherein the person re-identification model includes an encoder and a student decoder connected in sequence; The method of training the re-identification training model using the training set includes performing a second-stage training on the re-identification training model to obtain a pedestrian re-identification model. In each training of the second-stage training, the following steps are performed: The image samples of the training set are converted into training flattened sequences, and the training flattened sequences are added with class tokens; Input the training flattened sequence to the encoder to obtain pedestrian features; The mask generator generates a pedestrian part mask according to pedestrian features, and the pedestrian part mask includes sub-masks of p parts of the pedestrian; The teacher decoder decodes the feature area corresponding to the pedestrian part mask in the pedestrian feature to obtain the teacher pedestrian part feature, wherein the teacher pedestrian part feature includes the features of p parts of the pedestrian, where p is a positive integer; The student decoder decodes the pedestrian features to obtain the student pedestrian part features, wherein the student pedestrian part features include the features of p parts of the pedestrian; The total loss is calculated based on pedestrian features, pedestrian part masks, teacher pedestrian part features, student pedestrian part features, re-identification labels, and human body part parsing labels. The total loss includes pedestrian re-identification loss, part feature similarity loss, mask loss, and distillation loss. The back-propagation algorithm is used to update the network parameters of the re-identification training model according to the total loss.

7. A pedestrian re-identification model training method as claimed in claim 6, characterized in that: The re-ID training model also includes a distillation loss module for calculating the distillation loss for adaptive mask generation. The distillation loss module includes: The first multi-head self-attention layer performs multi-head self-attention processing on the student pedestrian part features to obtain the student self-attention features; An adaptive mask generation unit generates an adaptive mask based on the three-dimensional student self-attention features; A multiplication unit multiplies the adaptive mask with the three-dimensional student pedestrian part features to obtain the unmasked features of the pedestrian; The convolution unit performs convolution processing on the unmasked features of pedestrians to obtain the features of the reconstructed student pedestrian parts; The loss calculation unit calculates the adaptive mask generation distillation loss based on the reconstructed student pedestrian part features and the three-dimensional teacher pedestrian part features, and uses the adaptive mask generation distillation loss as the distillation loss.

8. The method for training a person re-identification model according to claim 7, characterized in that: The adaptive mask generation unit performs: Generate a mask template with the same size as the three-dimensional student self-attention feature; Traversing the feature points of the three-dimensional student self-attention feature, executing: calculating the difference between the feature value of the current traversed feature point and the feature threshold, if the difference is greater than the difference threshold, then setting the feature value of the feature point corresponding to the position of the current traversed feature point in the mask template to 1, if the difference is less than or equal to the difference threshold, then setting the feature value of the feature point corresponding to the position of the current traversed feature point in the mask template to 0; After traversing all the feature points of the three-dimensional student self-attention feature, the mask template with the updated feature values ​​of all feature points is used as the adaptive mask.

9. A person re-identification model training method according to any one of claims 6 to 8, characterized in that: The re-identification training model also includes a CLIP model, which uses a text encoder of the CLIP model to convert text descriptions into text visual features, and a learnable text token is embedded in the text description; The encoder is an adaptive multi-granularity feature fusion encoder, which includes a plurality of stacked Transformer encoding modules and one or more multi-granularity feature fusion modules; wherein each multi-granularity feature fusion module is located between two adjacent layers of Transformer encoding modules, and is used for weighted fusion of output features of all layers of Transformer encoding modules before the multi-granularity feature fusion module and a flattened sequence to obtain multi-granularity optimization features, and input the multi-granularity optimization features into a subsequent layer of Transformer encoding modules of the multi-granularity feature fusion module or output them as pedestrian features; Each multi-granularity feature fusion module includes: The text-visual feature interaction module inputs the query embedding and text-visual features learned in the person re-identification model training into the first multi-head attention layer for interactive calculation to obtain the first query feature; The query image interaction module concatenates the class tokens in the output features of the Transformer encoding modules of all layers before the multi-granularity feature fusion module and the class tokens in the flattened sequence to obtain a concatenated token, and inputs the first query feature and the concatenated token into the second multi-head attention layer for interactive calculation to obtain a second query feature; The feature weighting module uses the non-zero elements of the diagonal matrix of the second query feature as the fusion weights corresponding to the output features and flattened sequences of all layers of Transformer encoding modules before the multi-granularity feature fusion module in sequence, performs weighted processing on the output features and flattened sequences of all layers of Transformer encoding modules before the multi-granularity feature fusion module according to the fusion weights to obtain multi-granularity optimized features, and inputs the multi-granularity optimized features into the Transformer encoding module of the next layer of the multi-granularity feature fusion module or as pedestrian features.

10. The method for training a person re-identification model according to claim 9, characterized in that: The re-identification training model is trained by using the training set to first perform the first stage training and then perform the second stage training. The first stage training is to loop the following steps until the alignment loss satisfies the alignment loss condition: Input the text description with text TOKEN into the text encoder of CLIP model to obtain text visual features; Input the training flattened sequence corresponding to the sample image to the encoder to obtain pedestrian features; Calculate the alignment loss by using text visual features and pedestrian features; The text tokens are gradient updated according to the alignment loss.

Images