A pedestrian re-identification method based on multimodal auxiliary information-guided learning

Abstract

The application discloses a kind of pedestrian re-identification methods based on multi-modal auxiliary information guided learning, belong to computer vision pedestrian re-identification field, construct model MMFI to carry out pedestrian re-identification;Model MMFI includes multi-modal information extraction module, auxiliary information guided learning module, knowledge distillation network;Specifically include the following steps: step 1, obtain the RGB image of pedestrian as the input of multi-modal information extraction module, input image is generated by three different networks three kinds of modal features carrying different information;Step 2, construct auxiliary information guided learning module and implicit guide loss function;Step 3, construct knowledge distillation network and explicit guide loss function;Step 4, construct overall loss function, for model MMFI overall back propagation;Step 5, construct inference module, reduce the complexity of model MMFI.The method of the application is accurate and high in pedestrian re-identification, and high in computational efficiency.

Description

Technical Field

[0001] This invention belongs to the field of pedestrian re-identification in computer vision, specifically relating to a pedestrian re-identification method based on multimodal auxiliary information-guided learning. Background Technology

[0002] Deep learning-based person re-identification and detection techniques have made groundbreaking progress in recent years. These methods can be broadly categorized into three types: CNN-based methods, Transformer-based methods, and visual language model-based contrastive learning methods. CNN-based methods, such as the RPP method, improve accuracy by adaptively segmenting pedestrian regions and enhancing local feature representation. However, these methods can only process a portion of the image at a time, thus losing the connections between different blocks. Transformer-based methods, such as TransReID, aggregate global and local contextual information using the Transformer. The features extracted in this approach generally contain background and other interfering information; improper feature processing design can lead to insufficient model robustness. Furthermore, contrastive learning methods using visual language models can guide the image encoder to focus more on the pedestrian portion through text guidance. For example, the text generated by the CLIP-ReID method is insufficient to allow the image features to learn enough information about the pedestrian, thus failing to guide the image encoder to focus more on the pedestrian. Summary of the Invention

[0003] To address the aforementioned issues, this invention proposes a pedestrian re-identification method based on multimodal auxiliary information-guided learning, constructing a model MMFI for pedestrian re-identification. This method utilizes multiple feature extraction models to generate multimodal information, uses auxiliary information to guide image features to focus on pedestrians, and removes auxiliary information during the inference stage to maintain retrieval accuracy while reducing model complexity.

[0004] The technical solution of the present invention is as follows:

[0005] A pedestrian re-identification method based on multimodal auxiliary information-guided learning is proposed, which constructs a model MMFI for pedestrian re-identification. The MMFI model includes a multimodal information extraction module, an auxiliary information-guided learning module, and a knowledge distillation network. The specific steps include:

[0006] Step 1: Obtain the RGB image of the pedestrian as the input of the multimodal information extraction module. The input image is processed by three different networks to generate three modal features carrying different information.

[0007] Step 2: Construct the auxiliary information-guided learning module and the implicit guided loss function;

[0008] Step 3: Construct a knowledge distillation network and an explicit guided loss function;

[0009] Step 4: Construct the overall loss function for the overall backpropagation of the MMFI model;

[0010] Step 5: Build an inference module to reduce the complexity of the MMFI model.

[0011] Furthermore, in step 1, the multimodal information extraction module includes three different feature extraction networks for images, text, and key points, respectively extracting features for the three modalities of images, text, and key points; the specific process is as follows:

[0012] Step 1.1: The image feature extraction network first divides the input image into P small blocks, and then inputs them into the image encoder to extract the corresponding image features, which include global features. and local features Where B represents the batch size; D represents the channel dimension; and P represents the number of blocks.

[0013] Step 1.2: The text feature extraction network first feeds the input image into a pre-trained large language model to generate a text description. Then, it segments the text description into discrete units that the model can process, and inputs them into the text feature encoder of the CLIP multimodal model to generate text features.

[0014] Step 1.3: The key point feature extraction network uses a pose estimation model to generate key point features for the input image. Where K represents the number of key points, and the heatmap features of the overall image are obtained through global average pooling.

[0015] Further, in step 2, the auxiliary information-guided learning module includes a first feature fusion module and a second feature fusion module; both the first and second feature fusion modules include a cross-attention mechanism, a residual self-attention mechanism, and a residual feedforward network. The feedforward neural network is connected to the classification head, and the classification result is constrained by an implicit guided loss function; the specific process is as follows:

[0016] Step 2.1: The first feature fusion module uses the generated text features as global guiding features. It processes the global features of the image through cross-attention, residual self-attention, and residual feedforward networks to obtain global fused features, and adds batch normalization between each step; the formula is as follows:

[0017]

[0018] in, This represents the features generated by the cross-attention mechanism of global image features and text features; express Features obtained through residual self-attention; express The global fusion features are obtained through the residual feedforward network; BN(·) represents the batch normalization operation; Softmax(·) is the Softmax function; d represents the feature dimension; Q g Representing text features F T Through mapping function The query obtained; K g Representing global features of an image Through mapping function The obtained key; V g Representing global features of an image Through mapping function The value obtained; Representation of features Through mapping function The query returned; Representation of features Through mapping function The obtained key; Representation of features Through mapping function The obtained values; Linear(·) represents a linear layer; GELU(·) represents the activation function; T is the transpose;

[0019] Step 2.2: Segment the local features of the image to obtain K local sub-features, the same number as the number of keypoints. The second feature fusion module uses the generated keypoint features as local guiding features. It processes the local features of the image through cross-attention, residual self-attention, and residual feedforward networks to obtain the local fused features; the formula is as follows:

[0020]

[0021] in, This represents the feature generated by the cross-attention mechanism between local sub-features and keypoint features of an image; express Features obtained through residual self-attention; express Local fusion features obtained through the residual feedforward network; Q l Representing key point features F K Through mapping function The query obtained; K l Representing local sub-features Through mapping function The obtained key; V lRepresenting local sub-features Through mapping function The value obtained; Representation of features Through mapping function The query returned; Representation of features Through mapping function The obtained key; Representation of features Through mapping function The value obtained;

[0022] Step 2.3: Concatenate the global fusion features and the local fusion features, and feed the concatenated result into the classification head using the Softmax function to obtain the classification result. The implicit guided loss function uses entity loss and triplet loss to supervise the classification result; the formula is as follows:

[0023]

[0024] L Triplet =max(d p -d n +m,0) (13);

[0025] L Im =L ID +L Triplet (14);

[0026] Where [·,·] denotes the feature concatenation operation; S fus This indicates that the image scores for different pedestrian categories are calculated using the Softmax function, and the highest score is selected as the pedestrian category for the current image; y j This represents the true label of the j-th image; S represents the total number of images; j For the category result of the j-th image; L ID For entity loss; d p Indicates a positive sample; d n Represents a negative sample; m represents the boundary; L Triplet For triplet loss; L Im This represents implicit guided loss.

[0027] Furthermore, in step 3, the knowledge distillation network includes a teacher network and a student network. The teacher network guides the student model to learn information through knowledge distillation loss, and the student model learns features by constraining them through student loss. The specific process is as follows:

[0028] Step 3.1: The global features, text features, and heatmap features of the image are stitched together. The stitched features are then input into a feedforward network, a batch normalization layer, a linear layer, and an activation function to obtain teacher features. The global features of the image are used as student features. Knowledge distillation loss is applied to ensure that the student fully learns the information from the teacher. The formula is as follows:

[0029] F student =f I g (15);

[0030] F teacjer =BN(Linear(ReLU(Linear(BN([F I g ,F T ,F h ]))))) (16);

[0031] L KD =MSE(F student ,F teacher (17);

[0032] Among them, F teacher For teacher characteristics; F student For student characteristics; ReLU(·) represents the activation function; MSE(·) represents the mean squared error loss; L KD For knowledge distillation loss;

[0033] Step 3.2: For student characteristics, obtain the classification score S using a classification head employing the Softmax function. student Constraints are applied separately using student loss; finally, an explicit guided loss function is constructed; the formula is as follows:

[0034] L student =L ID +L Triplet (18);

[0035] L ex =L student +λL KD (19);

[0036] Among them, L student Losses to students; L ex λ represents the explicit guided loss function; λ is a hyperparameter.

[0037] Furthermore, in step 4, L Im With L ex Combined with the overall loss function L of the MMFI model total The formula is as follows:

[0038] Ltotal =L Im +L ex (20).

[0039] Furthermore, in step 5, in the inference module, the large language model and pose estimation model are removed, and only the image encoder is retained for the input query image I. query and images in the database I gallery Extract query features and database features separately; then calculate the distance similarity between the query features and database features using Euclidean distance, select the image with the smallest Euclidean distance as the final pedestrian image category, and compare whether the categories of this pedestrian image and the query image are equal. If they are equal, it indicates that the model performance is good, and the MMFI model is applied to pedestrian re-identification; otherwise, return to step 1 and repeat the process; the formula is as follows:

[0040] F query =Encoder(I query ) (twenty one);

[0041] F gallery =Encoder(I gallery ) (twenty two);

[0042]

[0043] Among them, F query Indicates query characteristics; F gallery Represents database features; Encoder(·) is the image encoder; I is the category of the final pedestrian image; This means finding the image in the database that has the smallest Euclidean distance to the query image; Let be the i-th database feature; G represents the total number of images in the database.

[0044] The beneficial technical effects of this invention are as follows: It utilizes three models to generate multimodal information containing different content; the information from other modalities can help supplement image information, making its learning of pedestrian information more comprehensive; implicit and explicit guidance allows image features to fully learn the additional information contained in the features of other modalities; removing the information of other modalities during retrieval can effectively reduce model complexity and improve model efficiency while ensuring that image features can learn additional information. The method of this invention has high pedestrian re-identification accuracy and high computational efficiency. Attached Figure Description

[0045] Figure 1 This is a framework diagram of cross-modal information interaction in this invention.

[0046] Figure 2 This is a flowchart of the reasoning module of the present invention. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0048] The method of this invention includes three processes: multimodal information extraction, auxiliary information-guided image feature extraction, and reduction of model complexity. Based on these three processes, a cross-modal information interaction framework and inference module are proposed, such as... Figure 1 and Figure 2 As shown, the multimodal feature extraction process consists of three feature extraction networks designed in this invention. Based on the input pedestrian image, image features representing global and local information, text features representing pedestrian semantic information, and key point features representing pedestrian structure are extracted respectively. The latter two are used as auxiliary information to guide image features to better focus on pedestrians and avoid confusion. The auxiliary information-guided image feature learning process is responsible for implicitly guiding the image features by fusing them with the extracted auxiliary information, and explicitly guiding them by using direct loss constraints. This enables image features to learn knowledge that they cannot learn on their own, improving the model's retrieval accuracy. The process of reducing model complexity refers to removing the auxiliary information and using the learned image features for retrieval during the retrieval stage. This maintains accuracy while speeding up the retrieval process and increasing model efficiency.

[0049] A pedestrian re-identification method based on multimodal auxiliary information-guided learning is proposed, which constructs a model MMFI for pedestrian re-identification. The MMFI model includes a multimodal information extraction module, an auxiliary information-guided learning module, and a knowledge distillation network. Loss functions are designed for the auxiliary information-guided learning module and the knowledge distillation network, and the two loss functions are added together to construct the overall loss function of the MMFI model, which is used for overall backpropagation of the model. Simultaneously, an inference module is designed to reduce the model complexity. The specific steps include:

[0050] Step 1: Obtain the RGB image of the pedestrian as input to the multimodal information extraction module. The input image is processed by three different networks to generate three modal features carrying different information. The multimodal information extraction module includes three different feature extraction networks for image, text, and keypoints, respectively extracting features for the image, text, and keypoint modalities. The specific process is as follows:

[0051] Step 1.1: The image feature extraction network first divides the input image into P small patches, and then inputs them into the image encoder to extract the corresponding image features, which include global features. and local features Where B represents the batch size; D represents the channel dimension; and P represents the number of blocks. Image features are used as the main features for backpropagation learning. The image encoder used in this invention is specifically VisionTransformer, which is a deep learning model for image classification based on the Transformer architecture.

[0052] Step 1.2: Benefiting from the powerful semantic understanding capabilities of the large language model, the text feature extraction network first feeds the input image into the pre-trained large language model to generate a text description. Then, it tokenizes the text description into discrete units (tokens) that the model can process, and inputs them into the text feature encoder of the CLIP (Contrastive Language–Image retraining) multimodal model to generate text features. The large language model used in this invention is specifically Multimodal LLM, which is a large-scale artificial intelligence model capable of simultaneously processing and understanding multiple types of data (such as text, images, audio, video, etc.).

[0053] Step 1.3: Key points can provide structural information about pedestrians. Therefore, the key point feature extraction network uses the pose estimation model HRNet to generate key point features from the input image. Where K represents the number of key points, and the heatmap features of the overall image are obtained through global average pooling.

[0054] Step 2: Construct an auxiliary information-guided learning module and an implicit guided loss function. The auxiliary information-guided learning module is used for implicit guided learning, and the implicit guided loss function constrains the implicit guided image feature learning. The auxiliary information-guided learning module includes a first feature fusion module and a second feature fusion module. Both the first and second feature fusion modules include a cross-attention mechanism, a residual self-attention mechanism, and a residual feedforward network. The feedforward neural network is followed by a classification head, and the classification result is constrained by the implicit guided loss function. The specific process is as follows:

[0055] Step 2.1: The first feature fusion module uses the generated text features as global guiding features. It processes the global features of the image through cross-attention, residual self-attention, and residual feedforward networks to obtain global fused features, and adds batch normalization between each step; the formula is as follows:

[0056]

[0057] in, This represents the features generated by the cross-attention mechanism of global image features and text features; express Features obtained through residual self-attention; express The global fusion features are obtained through the residual feedforward network; BN(·) represents the batch normalization operation; Softmax(·) is the Softmax function; d represents the feature dimension; Q g Representing text features F T Through mapping function The query obtained; K g Representing global features of an image Through mapping function The obtained key; V g Representing global features of an image Through mapping function The value obtained; Representation of features Through mapping function The query returned; Representation of features Through mapping function The obtained key; Representation of features Through mapping function The obtained values; Linear(·) represents a linear layer; GELU(·) represents the activation function; T is the transpose;

[0058] Step 2.2: Segment the local features of the image to obtain K local sub-features. The second feature fusion module uses the generated keypoint features as local guiding features. It processes the local features of the image through cross-attention, residual self-attention, and residual feedforward networks to obtain the local fused features. The formula is as follows:

[0059]

[0060] in, This represents the feature generated by the cross-attention mechanism between local sub-features and keypoint features of an image; express Features obtained through residual self-attention; express Local fusion features obtained through the residual feedforward network; Q l Representing key point features F K Through mapping function The query obtained; K l Representing local sub-features Through mapping function The obtained key; Vl Representing local sub-features Through mapping function The value obtained; Representation of features Through mapping function The query returned; Representation of features Through mapping function The obtained key; Representation of features Through mapping function The value obtained;

[0061] Step 2.3: Concatenate the global and local fusion features, and feed the concatenated result into a classification head using the Softmax function to obtain the classification result. The implicit guided loss function uses entity loss and triplet loss to supervise the classification result. The formula is as follows:

[0062]

[0063]

[0064] L Triplrt =max(d p -d n +m,0) (13);

[0065] L Im =L ID +L Triplet (14);

[0066] Where [·,·] denotes the feature concatenation operation; S fus This indicates that the image scores for different pedestrian categories are calculated using the Softmax function, and the highest score is selected as the pedestrian category for the current image; y j This represents the true label of the j-th image; S represents the total number of images; j For the category result of the j-th image; L ID For entity loss; d p Indicates a positive sample; d n Represents a negative sample; m represents the boundary; L Triplet For triplet loss; L Im This represents implicit guided loss.

[0067] Step 3: Construct a knowledge distillation network and simultaneously construct an explicit guided loss function to constrain the knowledge distillation network, achieving the effect of explicitly guiding image feature learning. The knowledge distillation network consists of a teacher network and a student network. The teacher network guides the student model to learn information through the knowledge distillation loss, and the student model itself also constrains feature learning through the student loss. The specific process is as follows:

[0068] Step 3.1: The global features, text features, and heatmap features of the image are stitched together. The stitched features are then input into a feedforward network, a batch normalization layer, a linear layer, and an activation function to obtain teacher features. The global features of the image are used as student features. Knowledge distillation loss is applied to ensure that students fully learn the information from the teacher. The formula is as follows:

[0069] F student =F I g (15);

[0070] F teacher =BN(Linear(ReLU(Linear(BN([F I g ,F T ,F h ]))))) (16);

[0071] L KD =MSE(F student ,F teacher (17);

[0072] Among them, F teacher For teacher characteristics; F student For student characteristics; ReLU(·) represents the activation function; MSE(·) represents the mean squared error loss; L KD This is due to knowledge distillation loss.

[0073] Step 3.2: For student characteristics, obtain the classification score S using a classification head employing the Softmax function. student The student loss is used as a separate constraint to measure the gap between the student's data and the true label; finally, an explicit guided loss function is constructed. The formula is as follows:

[0074] L student =L ID +L Triplet (18);

[0075] L ex =L student +λL KD (19);

[0076] Among them, L studentLosses to students; L ex λ represents the explicit guided loss function; λ is a hyperparameter used to constrain the scale of the knowledge distillation loss.

[0077] Step 4: Construct the overall loss function for the overall backpropagation of the MMFI model; L Im With L ex Combined into the overall loss function L total The formula is as follows:

[0078] L total =L Im +L ex (20);

[0079] Step 5: Build the inference module, remove the two auxiliary networks, and reduce the complexity of the MMFI model.

[0080] Introducing multiple models into auxiliary networks significantly increases inference complexity, making algorithm deployment and implementation difficult. Therefore, this invention investigates a method for removing auxiliary information during inference and using only image features for retrieval. The specific process is as follows:

[0081] In the inference module, the large language model and pose estimation model are removed, and only the image encoder is retained. For the input query image I... query and images in the database I gallery Extract query features and database features separately; then calculate the distance similarity between the query features and database features using Euclidean distance, and select the image with the smallest Euclidean distance as the final pedestrian image category. Compare whether the categories of this pedestrian image and the query image are equal. If they are equal, it indicates that the model performance is good, and the MMFI model can be applied to pedestrian re-identification; otherwise, return to step 1 and process again. The formula is as follows:

[0082] F query =Encoder(I query ) (twenty one);

[0083] F gallery =Encoder(I gallery ) (twenty two);

[0084]

[0085] Among them, F query Indicates query characteristics; F gallery Represents database features; Encoder(·) is the image encoder; I is the category of the final pedestrian image; This means finding the image in the database that is closest to the query image; Let be the i-th database feature; G represents the total number of images in the database.

[0086] To demonstrate the feasibility and superiority of this invention, the following comparative experiments were conducted.

[0087] Comparative Experiment 1:

[0088] Comparative Experiment 1 evaluates the proposed MMFI model on two publicly available and challenging person re-identification datasets: Market-1501 and DukeMTMC-ReID. Market-1501 is a widely used person re-identification dataset, covering 1501 pedestrians captured by 6 cameras; the training set contains 751 pedestrians with a total of 12936 images, and the test set contains 750 pedestrians with a total of 19732 images. DukeMTMC-ReID is a large-scale multi-target, multi-camera pedestrian tracking dataset recorded by 8 simultaneous cameras, covering 1812 pedestrians; the training set contains 702 pedestrians with a total of 16522 images, and the test set contains 1110 pedestrians with a total of 17661 images.

[0089] In Comparative Experiment 1, the Rank-1 accuracy and mean precision (AP) of each image in the query set were calculated, followed by the mean AP (mAP) for the entire query set. The training parameters for all datasets were: 300 training epochs, the AdamW optimizer, an initial learning rate of 0.0001 (decreasing by 0.1 every 10 epochs), and a dropout rate of 0.5. Each input image was resized to 256×128 for both training and testing, with padding used for enlarged portions. Data augmentation was performed through random scaling, random cropping, and horizontal flipping. The hyperparameter of the method in this invention was set to the distillation loss parameter λ = 0.5.

[0090] The MMFI model of this invention was compared with 15 other models—TransReID, PFD, FED, DACL, CLIP-ReID, DC-Former, SAP, PHA, FPC, ADP, DPM-SPT, CCAFL, ProFD, OAT, and FusionReID—on the Market-1501 and DukeMTMC-ReID datasets. TransReID is the first to use the Vision Transformer as its feature extraction model and has established a strong baseline based on it. PFD utilizes pose information to explicitly separate semantic components and selectively matches unoccluded parts. FED aims to enhance the model's ability to identify target pedestrians in pedestrian re-identification tasks by simulating non-pedestrian occlusion and non-target pedestrians. DACL enhances the interaction between the global image and local regions, as well as image pairs, through global-local cross-attention and pairwise cross-attention mechanisms. CLIP-ReID utilizes a visual model initialized from the CLIP model and employs a two-stage training strategy, combining learnable text tags and a text encoder to generate fuzzy descriptions, thereby enhancing the model's fine-grained visual classification capabilities. DC-Former proposes a novel Transformer network architecture that increases identity density by creating multiple diverse and compact subspaces within the embedding space. SAP combines a semantic partitioning teacher model and an attention-based student model, utilizing knowledge distillation to align the attention partitioning with the semantic partitioning. PHA enhances the representation of high-frequency features through discrete high-frequency component enhancement and patch-level contrastive loss. FPC removes background noise and occlusion-related interference features using a sparse encoder and recovers lost target information from the image library using multi-view feature matching and feature integration modules. ADP simulates real-world occlusion by introducing an attention-based interference mask and simultaneously processes complete and occluded images using a dual-path constraint module. DPM-SPT divides images into identity and occlusion sets using an image encoder and generates realistic occlusion samples through attention-guided patch transfer. CCAFL utilizes text information generated by CLIP for semantic supervision and combines intra-camera discriminative learning and inter-camera adversarial learning to learn robust features across cameras. ProFD leverages CLIP's pre-trained text modality knowledge to design text cues for specific body parts, and combines a hybrid attention decoder and a self-distillation strategy to generate well-aligned human body part features. OAT introduces a second-order attention module and an entropy-guided fusion module to capture higher-order relationships between different parts or regions of a pedestrian and mitigate uncertainties caused by occlusion, respectively. FusionReID uses a dual-branch feature extraction and dual-attention fusion module to deeply fuse local features extracted by a convolutional neural network with global features extracted by a Transformer.The comparison results are shown in Table 1.

[0091] Table 1 shows the differences between this invention and other models on the Market-1501 and DukeMTMC-ReID datasets.

[0092]

[0093] As shown in Table 1, the MMFI model of this invention achieved mean accuracy (mAP) of 93.2% and 85.4% on the Market-1501 and DukeMTMC-ReID datasets, respectively, with Rank-1 accuracy of 96.0% and 90.9%. These results clearly demonstrate the effectiveness of the MMFI framework proposed in this invention for pedestrian re-identification tasks.

[0094] Comparative Experiment 2:

[0095] To effectively reduce model complexity, this invention uses only image features for retrieval. Therefore, while maintaining accuracy, this model was compared with TransReID, PFD, DC-Former, and PHA models in terms of frames per second (FPS), number of model parameters, and inference time. The comparison results are shown in Table 2.

[0096] Table 2 compares the frames per second, number of model parameters, and inference time of this invention with other models.

[0097]

[0098] As shown in Table 2, the model of this invention uses only pedestrian image features for retrieval during the inference stage, effectively improving the frames per second (FPS) and reducing the number of parameters and inference time. This demonstrates that the model of this invention can effectively improve model efficiency and reduce model computational complexity while ensuring recognition accuracy.

[0099] The experimental results demonstrate the effectiveness of the method described in this invention in significantly improving the average class accuracy (mean value) in challenging person re-identification benchmarks, including Market-1501 and DukeMTMC-ReID. Compared to previous unimodal and multimodal learning models, it achieves a significant improvement of over 1%.

[0100] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A pedestrian re-identification method based on multi-modal auxiliary information guided learning, characterized in that, The model MMFI is constructed for pedestrian re-identification; the model MMFI comprises a multi-modal information extraction module, an auxiliary information guided learning module and a knowledge distillation network; and specifically comprises the following steps: Step 1, obtaining an RGB image of a pedestrian as input of the multi-modal information extraction module, the input image is generated through three different networks to generate three modal features carrying different information; Step 2, constructing an auxiliary information guided learning module and an implicit guided loss function; In step 2, the auxiliary information guided learning module comprises a first feature fusion module and a second feature fusion module; Both the first feature fusion module and the second feature fusion module comprise a cross-attention mechanism, a residual self-attention mechanism and a residual feedforward network, and a classification head is connected after the feedforward neural network, and the classification result is constrained by using the implicit guided loss function; the specific process is as follows: Step 2.1, the first feature fusion module takes the generated text feature as a global guided feature, and processes the global feature of the image through cross-attention, residual self-attention and residual feedforward network to obtain a global fusion feature, and batch normalization operation is added between each step; the formula is as follows: （1）； , , （2）； （3）； , , （4）； （5）； wherein, denotes the feature generated by the cross-attention mechanism from the image global feature and the text feature, denotes the batch size, denotes the channel dimension; denotes the feature obtained by the residual self-attention; denotes the global fusion feature obtained by the residual feedforward network; denotes the batch normalization operation; is the Softmax function; denotes the feature dimension; denotes the text feature obtained by the mapping function the query; denotes the image global feature obtained by the mapping function the key; denotes the image global feature obtained by the mapping function the value; denotes the feature obtained by the mapping function the query; denotes the feature obtained by the mapping function the key; denotes the feature obtained by the mapping function the value; denotes the linear layer; denotes the activation function; is the transpose; Step 2.2, the local features of the image are segmented to obtain the same number of local sub-features as the key points The second feature fusion module takes the generated key point features as local guide features, processes the local features of the image through cross attention, residual self-attention and residual feedforward network, and obtains local fusion features; the formula is as follows:​ （6）； , , （7）； （8）； , , （9）； （10）； wherein, denotes the feature generated by the cross attention mechanism between the local sub-feature and the keypoint feature, denotes the number of keypoint; denotes the feature obtained by the residual self-attention; denotes the local fusion feature obtained by the residual feedforward network; denotes the keypoint feature the query obtained by the mapping function ; denotes the local sub-feature the key obtained by the mapping function ; denotes the local sub-feature the value obtained by the mapping function ; denotes the feature the query obtained by the mapping function ; denotes the feature the key obtained by the mapping function ; denotes the feature the value obtained by the mapping function ; Step 2.3, the global fusion feature and the local fusion feature are spliced, and the spliced result is sent into a classification head using a Softmax function to obtain a classification result, and the implicit guided loss function uses entity loss and triplet loss to supervise the classification result; the formula is as follows: （11）； （12）； （13）； （14）； wherein, represents a feature concatenation operation; represents calculating the scores of different human categories of the image pair by a Softmax function, and selecting the highest score as the human category of the current image; represents the true label of the th image; is the total number of images; is the category result of the th image; is the entity loss; represents a positive sample; represents a negative sample; represents a boundary; is the triple loss; represents an implicit guidance loss; Step 3, constructing a knowledge distillation network and an explicit guided loss function; Step 4, constructing an overall loss function for overall back propagation of the model MMFI; Step 5, constructing an inference module to reduce the complexity of the model MMFI.

2. The method of claim 1, wherein the multi-modal auxiliary information guided learning based pedestrian re-identification method is characterized by, In step 1, the multi-modal information extraction module comprises three different feature extraction networks of image, text and key points, which correspond to the features of three modalities of image, text and key points respectively; the specific process is as follows: Step 1.

1. The image feature extraction network first divides the input image into small patches, and then inputs them into the image encoder to extract the corresponding image features, which include global features and local features , where denotes the batch size; denotes the channel dimension; denotes the number of small patches. Step 1.2, the text feature extraction network first sends the input image into a pre-trained large language model to generate a text description, and then divides the text description into discrete units that can be processed by the model and inputs them into a text feature encoder of a CLIP multi-modal model to generate text features ; Step 1.3, the key point feature extraction network uses a pose estimation model to generate key point features for the input image wherein represents the number of key points, and a heat map feature of the overall image is obtained through global average pooling .

3. The method of claim 2, wherein the multi-modal auxiliary information guided learning based pedestrian re-identification method is characterized by, In step 3, the knowledge distillation network comprises a teacher network and a student network, the teacher network guides the student model to learn information through a knowledge distillation loss, and the student model constrains feature learning through a student loss; The specific process is as follows: Step 3.1, splicing the global feature of the image, the text feature and the heat map feature, inputting the spliced feature into a feedforward network, a batch normalization layer, a linear layer and an activation function to obtain a teacher feature, taking the global feature of the image as a student feature, and making the student end fully learn the information of the teacher end through a knowledge distillation loss; the formula is as follows: （15）； （16）； （17）； wherein, is a teacher feature; is a student feature; denotes an activation function; denotes a mean squared error loss; is a knowledge distillation loss; Step 3.2, for student features, get classification scores through a classification head with Softmax function , constraints are made separately by student loss; finally, an explicit guidance loss function is constructed; the formula is as follows: （18）； （19）； wherein, is the student loss; denotes the explicit guidance loss function; is a hyperparameter.

4. The method of claim 3, wherein the multi-modal auxiliary information guided learning based pedestrian re-identification method is characterized by, In step 4, the With The overall loss function of the model MMFI is constructed in combination , as follows: （20）。 5. The method of claim 4, wherein the multi-modal auxiliary information guided learning based pedestrian re-identification method is characterized by, In step 5, in the reasoning module, the large language model is removed from the pose estimation model, and only the image encoder is retained for the input query image and the image in the database query features and database features are extracted respectively; Then, the query feature and the database feature are used to calculate the distance similarity by using the Euclidean distance, the image with the minimum Euclidean distance is selected as the final class of the pedestrian image, and whether the pedestrian image and the query image are equal is compared, if equal, it indicates that the model has good performance, and the model MMFI is applied to pedestrian re-identification; otherwise, return to step 1 for processing; the formula is as follows: （21）； （22）； （23）； wherein, represents a query feature; represents a database feature; is an image encoder; is a class of the final pedestrian image; represents finding an image in the database with the smallest Euclidean distance to the query image; is the th database feature; represents the total number of images in the database.

Images