A multimodal object re-identification method based on cue learning

Abstract

The present invention discloses a multimodal target re-identification method based on prompt learning, which relates to the field of computer vision and includes the following steps: constructing a multimodal pedestrian re-identification model based on prompt learning; establishing a loss function to perform model training on the multimodal pedestrian re-identification model based on prompt learning; applying the trained multimodal pedestrian re-identification model based on prompt learning to an actual scene to complete multimodal target re-identification based on prompt learning. The present invention solves the problems of high computational cost of existing multimodal re-identification methods and the significant distribution difference between different spectra making multimodal feature fusion more difficult.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a multimodal target re-identification method based on prompt learning. Background Art

[0002] The object re-identification (ReID) task has become an important research direction in the field of image recognition. Its main goal is to identify and match the same object by analyzing images taken by different cameras. Traditional ReID methods mostly rely on single-modality images, usually visible light (RGB) images, which work better under good lighting conditions. However, in environments with insufficient or large changes in lighting, the performance of RGB cameras will drop significantly, while near-infrared (NIR) and thermal infrared (TIR) ​​cameras can provide more stable image information.

[0003] Multimodal target re-identification can better cope with complex real-world scenarios by combining multiple spectral (modal) information. For example, in surveillance cameras, infrared cameras can capture clear images at night or in low-light environments, while RGB cameras are relied on during the day. Therefore, multimodal re-identification technology can not only improve recognition accuracy, but also enhance the robustness and practicality of the system.

[0004] Existing multimodal re-ID methods usually use complex, highly coupled architectures, which are effective but computationally expensive. In addition, the significant distribution differences between different spectra also make it difficult to fuse multimodal features. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multimodal target re-identification method based on prompt learning, which solves the problems of high computational cost of existing multimodal re-identification methods and the significant distribution differences between different spectra making multimodal feature fusion more difficult.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a multimodal target re-identification method based on prompt learning, comprising the following steps:

[0007] S1: Construct a multimodal person re-identification model based on cue learning;

[0008] S2: Establish a loss function to train the multimodal person re-identification model based on cue learning;

[0009] S3: Apply the trained cue-based multimodal person re-identification model to actual scenarios to complete cue-based multimodal target re-identification.

[0010] Furthermore, the multimodal person re-identification model based on prompt learning in S1 includes:

[0011] A modality feature extraction module, wherein the modality feature extraction module is used to extract global feature information from a multimodal image;

[0012] A modality hint vector module, which is used to establish a communication channel between different modality features, and to capture and transfer modality complementary information;

[0013] The modal feature fusion module is used to realize the interaction and fusion of different modal features;

[0014] The modality missing scenario adaptation module is used to dynamically adjust the model structure.

[0015] Further, the modality feature extraction module comprises a block encoder and a multi-layer serial self-attention feature extractor in sequence;

[0016] The block encoder is represented as:

[0017]

[0018]

[0019] in, is the visible light modality block coding sequence, is a visible light image, is a block encoder, is the visible light modal characteristic sequence, Encode the class, is the vector concatenation operation, Encode for position;

[0020]

[0021]

[0022] in, is the near infrared modal block coding sequence, is a near-infrared image, is the near-infrared modal feature sequence;

[0023]

[0024]

[0025] Among them, the thermal infrared modality block coding sequence is: For thermal infrared images, is the thermal infrared modal feature sequence;

[0026] The self-attention feature extractor includes a serial multi-head self-attention module, a feedforward neural network and a layer normalization module, which is expressed as:

[0027]

[0028]

[0029] in, For the The visible light feature sequence output by the multi-head attention layer in the layer self-attention feature extractor, is the self-attention feature extractor The visible light image input of the layer, is a multi-head self-attention module, is a feed-forward neural network, is the layer normalization module;

[0030]

[0031]

[0032] in, For the The near-infrared feature sequence output by the multi-head attention layer in the layer self-attention feature extractor, is the self-attention feature extractor Layer near infrared image input;

[0033]

[0034]

[0035] in, For the The thermal infrared feature sequence output by the multi-head attention layer in the layer self-attention feature extractor, is the self-attention feature extractor Thermal infrared image input of the layer.

[0036] Further, the modal prompt vector module includes a modal prompt vector injection module and a prompt exchange module;

[0037] The modality hint vector injection module introduces a modality hint vector into the input space of the modality feature extraction module, wherein the modality hint vector It is expressed as:

[0038]

[0039] in, is a learnable vector, is the dimension of the feature vector;

[0040] The modal prompt vector The vector cube formed by stacking the feature vectors in the block encoder is concatenated into , complete the injection of modal prompt vector;

[0041]

[0042]

[0043] in, is the vector cube, Indicates the operation of stacking tensors. is the set of real numbers, is the length of the feature vector;

[0044] The prompt exchange module is located at the output layer of the self-attention feature extractor and changes the position of the modality prompt vector by transposing the operation, which is expressed as:

[0045]

[0046] in, is the self-attention feature extractor The vector cube of the layer, is the self-attention feature extractor The modal hint vector of the layer, superscript Indicates that Transpose along the first and second dimensions.

[0047] Further, the modal feature fusion module includes a modal feature high response area selection module and a multimodal feature fusion module based on a modal prompt vector connected in series, and the modal feature high response area selection module is serially connected to the self-attention feature extractor in the modal feature extraction module;

[0048] The modal feature high response region selection module selects high response feature vectors from two perspectives: inter-modal vector selection and intra-modal vector selection. The inter-modal vector selection is expressed as:

[0049]

[0050] in, is the modal key area mask, To generate a key region mask based on the element index, To find the top ranked element from a set of elements Elements of and are the attention scores associated with the modality cue, respectively;

[0051] The modal internal vector selection is expressed as:

[0052]

[0053] in, is the intramodal key area mask, Attention scores for class encoding and image feature encoding;

[0054] The joint mask selected by the modality feature high response region selection module It is expressed as:

[0055]

[0056] in, Represents a union operation.

[0057] Furthermore, the multimodal feature fusion module based on the modal cue vector uses the modal cue vector as a query vector and guides the fusion of features of different modalities through a cross-attention mechanism. The formula of the cross-attention mechanism is:

[0058]

[0059]

[0060] in, is the cross attention layer, For the layer hint vector, For , and In the example, the high-response local feature vector sequence selected by the modal feature high-response region selection module is is the normalized exponential function, , and are the mapping matrices of query, keyword and value respectively, is the tensor dot product operator.

[0061] Furthermore, the fusion layer of the multimodal feature fusion module based on modal cue vector is expressed as:

[0062]

[0063]

[0064]

[0065] in, For the The cross-attention layer output feature sequence in the layer feature fusion module, is a multi-layer perceptron, is the final modal fusion feature, It is the feature sequence output by the feature fusion module of the last layer.

[0066] Further, the modality missing scene adaptation module includes a prompt vector shielding module and a feature sequence adjustment module, and the prompt vector shielding module and the feature sequence adjustment module are located between the block encoder and the first layer self-attention feature extractor;

[0067] The prompt vector masking module adjusts the structure and operation of the model through prompts related to the missing modality when the modality is detected to be missing;

[0068] The feature sequence adjustment module adjusts the multimodal token sequence in the case of modality loss.

[0069] Furthermore, the loss function in S2 is Combining triplet loss and cross entropy loss , the formula is:

[0070]

[0071]

[0072]

[0073] in, is the feature distance between the anchor sample and the positive sample, is the feature distance between the anchor sample and the negative sample, is a soft margin hyperparameter, For Category The smoothing label of The category predicted by the model The probability of is the number of categories.

[0074] The beneficial effects of the present invention are:

[0075] (1) The Prompt Transpose Mechanism and Prompt Modal Fusion Module (PBMF) proposed in this paper can effectively interact and fuse information between different modalities. By introducing the modal prompt vector into the VisionTransformer (ViT) architecture, the model can capture and utilize the complementary feature information between the modalities, significantly improving the expressive power of multimodal feature representation.

[0076] (2) By designing a flexible prompt masking and prompt transposition mechanism, the present invention can still maintain high performance when some modalities are missing. This ability to handle modal missingness makes the model more robust in practical application scenarios and is not prone to significant performance degradation due to partial sensor failure or data missing.

[0077] (3) The shared ViT architecture and cue-based feature fusion strategy adopted in this invention significantly reduce the number of model parameters and computational complexity while maintaining high performance. Compared with traditional methods, the use of a modality-sharing backbone network reduces hardware requirements and energy consumption.

[0078] (4) The mean average precision (mAP) and rank 1 accuracy of the multimodal re-ID model of the present invention are better than the current state-of-the-art methods on multiple benchmark datasets (such as MSVR310, RGBNT201, RGBNT100, Market1501MM, etc.). These experiments verify the efficiency and superiority of the present invention in handling multimodal re-ID tasks.

[0079] (5) Since the model has the ability to handle modality loss and maintain high performance under different modality combinations, the present invention can adapt to a variety of practical application scenarios, such as intelligent monitoring, autonomous driving, and identity authentication, in which the sensor types and acquisition conditions may change. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 Flowchart of a multimodal object re-identification method based on cue learning.

[0081] Figure 2 Figure 2: The structure diagram of the multimodal person re-identification model based on cue learning.

[0082] Figure 3 This is the structural diagram of the adaptive module for modality missing scenarios. DETAILED DESCRIPTION

[0083] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0084] like Figure 1 As shown, a multimodal target re-identification method based on prompt learning includes the following steps:

[0085] S1: Construct a multimodal person re-identification model based on cue learning;

[0086] S2: Establish a loss function to train the multimodal person re-identification model based on cue learning;

[0087] S3: Apply the trained cue-based multimodal person re-identification model to actual scenarios to complete cue-based multimodal target re-identification.

[0088] The present invention proposes a prompt-based multimodal feature fusion framework. By introducing a modal prompt exchange mechanism, the modal prompt vector can interact with the feature vectors of different modalities, establish a modal complementary information transmission channel, and achieve a more powerful multimodal feature representation by utilizing modal complementary information. Furthermore, the feature fusion process is optimized through the prompt selection module (Prompt-based Token Selection, PBTS) and the prompt modal fusion module (Prompt-based Modal Fusion, PBMF), the background interference is reduced, and the performance on multiple benchmark data sets is significantly improved. At the same time, the method can flexibly deal with the situation of modal missing, ensuring robustness and practicality in various complex environments.

[0089] like Figure 2 As shown, the multimodal person re-identification model based on prompt learning in S1 includes:

[0090] A modality feature extraction module, wherein the modality feature extraction module is used to extract global feature information from a multimodal image;

[0091] A modality hint vector module, which is used to establish a communication channel between different modality features, and to capture and transfer modality complementary information;

[0092] The modal feature fusion module is used to realize the interaction and fusion of different modal features;

[0093] The modality missing scenario adaptation module is used to dynamically adjust the model structure.

[0094] The modality feature extraction module includes a block encoder and a multi-layer serial self-attention feature extractor in sequence;

[0095] In this embodiment, the block encoder uses 16×16 convolution blocks (patches) with a step size of 16 to divide the input image into fixed-size 16×16 image blocks. These image blocks are then converted into patch beddings. This method flattens the two-dimensional spatial information of the image into a one-dimensional feature vector, allowing the model to process image data in a sequential manner. In order to enable the model to distinguish image blocks at different positions, a positional encoding is added to each image block vector. Positional encoding and block encoding are both vectors of the same dimension. By adding these two vectors, position information can be provided to the model while retaining the image block features. This helps the model understand the spatial relationship in the image. In addition to block encoding, a class token is introduced to represent the global features of the entire image. This class encoding vector is concatenated with the image block vector into an encoding sequence for further processing.

[0096] The block encoder is represented as:

[0097]

[0098]

[0099] in, is the visible light modality block coding sequence, is a visible light image, is a block encoder, is the visible light modal characteristic sequence, Encode the class, is the vector concatenation operation, Encode for position;

[0100]

[0101]

[0102] in, is the near infrared modal block coding sequence, is a near-infrared image, is the near-infrared modal feature sequence;

[0103]

[0104]

[0105] in, is the thermal infrared modality block encoding sequence, For thermal infrared images, is the thermal infrared modal feature sequence;

[0106] The encoded sequence is fed into multiple layers of self-attention feature extractors in series, each of which consists of a serial multi-head self-attention module, a feed-forward neural network, and layer normalization. The multi-head self-attention module enables the model to focus on different parts of the input sequence, thereby capturing the global and local features of the image. The feed-forward neural network further processes and transforms the features output by the self-attention mechanism. Layer Normalization: In each layer of the self-attention encoder, the layer normalization module is applied to the input data and residual connection to ensure that the input distribution of each layer is stable during forward propagation, which helps to accelerate the convergence of the model and improve the stability of training.

[0107] The self-attention feature extractor includes a serial multi-head self-attention module, a feedforward neural network and a layer normalization module, which is expressed as:

[0108]

[0109]

[0110] in, For the The visible light feature sequence output by the multi-head attention layer in the layer self-attention feature extractor, is the self-attention feature extractor The visible light image input of the layer, is a multi-head self-attention module, is a feed-forward neural network, is the layer normalization module;

[0111]

[0112]

[0113] in, For the The near-infrared feature sequence output by the multi-head attention layer in the layer self-attention feature extractor, is the self-attention feature extractor Layer near infrared image input;

[0114]

[0115]

[0116] in, For the The thermal infrared feature sequence output by the multi-head attention layer in the layer self-attention feature extractor, is the self-attention feature extractor Thermal infrared image input of the layer.

[0117] The modal prompt vector module includes a modal prompt vector injection module and a prompt exchange module;

[0118] The modality hint vector injection module introduces a modality hint vector into the input space of the modality feature extraction module, wherein the modality hint vector It is expressed as:

[0119]

[0120] in, is a learnable vector, is the dimension of the feature vector;

[0121] The modal prompt vector The vector cube formed by stacking the feature vectors in the block encoder is concatenated into , complete the injection of modal prompt vector;

[0122] After the block encoder in the multimodal feature extraction module encodes the image into a feature vector, the multimodal feature vectors (Token Embeddings) can be stacked into a vector cube:

[0123]

[0124]

[0125] in, is the vector cube, Indicates the operation of stacking tensors. is the set of real numbers, is the length of the feature vector;

[0126] The prompt exchange module is located at the output layer of the self-attention feature extractor and changes the position of the modality prompt vector by transposing the operation, which is expressed as:

[0127]

[0128] in, is the self-attention feature extractor The vector cube of the layer, is the self-attention feature extractor The modal hint vector of the layer, superscript Indicates that Transpose along the first and second dimensions.

[0129] The modal feature fusion module includes a modal feature high response area selection module and a multimodal feature fusion module based on a modal prompt vector, and the modal feature high response area selection module is serially connected to the self-attention feature extractor in the modal feature extraction module;

[0130] The modal feature high response region selection module selects high response feature vectors from two perspectives: Cross-modal Token Selection (CMTS) and Inner-Modal Token Selection (IMTS). The cross-modal vector selection is expressed as:

[0131]

[0132] in, is the modal key area mask, To generate a key region mask based on the element index, To find the top ranked element from a set of elements Elements of and are the attention scores associated with the modality cues, respectively (the attention scores are obtained through the multi-head self-attention module in the multimodal feature extraction module);

[0133] The modal internal vector selection is expressed as:

[0134]

[0135] in, is the intramodal key area mask, Attention scores for class encoding and image feature encoding;

[0136] The working mechanism of the modality feature high response area selection module (PBTS) is based on the modality cues introduced in the modality feature extraction module, accumulating and using normalized attention scores to select key feature vectors. By combining the masks of CMTS and IMTS, PBTS can comprehensively consider the high response areas between and within modalities to achieve more effective feature fusion;

[0137] The joint mask selected by the modality feature high response region selection module It is expressed as:

[0138]

[0139] in, Represents a union operation.

[0140] The main goal of the multimodal feature fusion module (PBMF) based on modal cue vector is to achieve effective fusion of features from different modalities. The PBMF module uses the modal cue vector as the query vector and guides the fusion of features from different modalities through the cross-attention mechanism. This enables the model to keep its focus on key areas during the fusion process, thereby improving the overall recognition performance. Specifically, in the cross-attention mechanism:

[0141] Query: The vector from the modality prompt is used as the query vector.

[0142] Key and Value: High response regions from the multimodal feature vector, which are selected by the previous PBTS module.

[0143] The multimodal feature fusion module based on modal cue vector uses the modal cue vector as a query vector and guides the fusion of features of different modalities through a cross-attention mechanism. The formula of the cross-attention mechanism is:

[0144]

[0145]

[0146] in, is the cross attention layer, For the layer hint vector, For , and In the example, the high-response local feature vector sequence selected by the modal feature high-response region selection module is is the normalized exponential function, , and are the mapping matrices of query, keyword and value respectively, is the tensor dot product operator.

[0147] The fusion layer in the PBMF module is similar to the image encoding layer in ViT, consisting of a cross attention layer (CA), a multi-layer perceptron (MLP), and a layer normalization (LN), and is equipped with a residual connection (ResidualConnection).

[0148] The fusion layer of the multimodal feature fusion module based on modal cue vector is expressed as:

[0149]

[0150]

[0151]

[0152] in, For the The cross-attention layer output feature sequence in the layer feature fusion module, is a multi-layer perceptron, is the final modal fusion feature, It is the feature sequence output by the feature fusion module of the last layer.

[0153] The modality missing scene adaptation module includes a prompt vector shielding module and a feature sequence adjustment module, wherein the prompt vector shielding module and the feature sequence adjustment module are located between the block encoder and the first layer self-attention feature extractor;

[0154] The prompt vector masking module adjusts the structure and operation of the model through prompts related to the missing modality when the modality is detected to be missing;

[0155] Specifically, some of the prompts in the modality prompt vector matrix are masked to avoid unnecessary calculations in the feature path of the missing modality. This prompt masking mechanism ensures that the prompt transposition operation can still be performed normally in the case of missing modalities without errors due to the mismatch between prompts and input features;

[0156] The feature sequence adjustment module adjusts the multimodal token sequence in the case of modality loss.

[0157] For example, when the NIR modality is missing, only feature sequences from the RGB and TIR modalities are used, which are stacked to form a new token cube and concatenated with the adjusted prompt sequence to form part of the input space of the multimodal feature extraction module, as shown in Figure 3 As shown, the formula is as follows:

[0158]

[0159]

[0160] in, For the new token sequence cube, is the adjusted prompt matrix.

[0161] In the case of missing modalities, the model's forward propagation process is adjusted accordingly to ensure that cue transposition and feature fusion can be performed effectively. Specifically, by masking the cues related to the missing modality, the model can still perform effective cue transposition operations and modality feature interactions in other modality paths.

[0162] In order to effectively train the multimodal object re-identification model, a loss function module combining triplet loss and cross-entropy loss is designed. Triplet loss is used to optimize the relative distance between samples, ensuring that the distance between samples of the same identity is closer and the distance between samples of different identities is farther. This loss function consists of an anchor sample, a positive sample, and a negative sample. By maximizing the distance difference between positive and negative samples, triplet loss can significantly improve the model's discriminative ability. Cross-entropy loss is used for classification tasks, and the loss is calculated by comparing the difference between the model's output and the true category label. It encourages the model to assign a higher probability to the correct category, thereby optimizing the model's classification ability. In the present invention, cross-entropy loss with label smoothing is used to prevent overfitting and improve generalization performance. The design of this module aims to simultaneously optimize the model's discriminative ability and classification performance, thereby improving the accuracy of multimodal object re-identification.

[0163] The loss function in S2 Combining triplet loss and cross entropy loss , the formula is:

[0164]

[0165]

[0166]

[0167] in, is the feature distance between the anchor sample and the positive sample, is the feature distance between the anchor sample and the negative sample, is a soft margin hyperparameter, For Category The smoothing label of The category predicted by the model The probability of is the number of categories.

[0168] In one embodiment of the present invention, the training of the model includes the following steps:

[0169] (1) Data preprocessing

[0170] Input image size: All input images are uniformly resized to 256 × 128 × 3. This is a common size choice that balances computational efficiency and preservation of image details.

[0171] Data enhancement: During the training process, some data enhancement techniques are used to improve the generalization ability of the model, including random cropping, horizontal flipping, and random erasing. These techniques can effectively increase the diversity of data and prevent the model from overfitting.

[0172] (2) Model architecture initialization

[0173] Modality feature extraction module: uses the ViT-Base-16 architecture, which contains 12 self-attention feature extractor layers, and the hidden size of each layer is 768 dimensions.

[0174] (3) Training hyperparameter setting

[0175] Batch size: set to 32. Each batch contains 4 groups of multispectral images, each group corresponds to 8 different identities.

[0176] Optimizer: Use Adam optimizer, which is a commonly used gradient descent optimization method with good convergence and stability.

[0177] Learning rate: The initial learning rate is set to 1×10^4 and decays to 0.1 and 0.01 at the 30th and 50th epochs.

[0178] (4) Loss function

[0179] A total loss function combining triplet loss and cross entropy loss is used. Triplet loss is used to optimize the relative distance between samples, while cross entropy loss is used to optimize classification performance. During training, these losses act on multimodal fusion features. .

[0180] (5) Training process

[0181] Epochs: The model is usually trained for 60 epochs, and the timing of early stopping is determined based on the performance of the validation set.

[0182] Forward propagation: For each input multimodal image, extract features and perform feature interaction and fusion through the modality cue vector. Calculate the loss and perform backpropagation to update the model parameters.

[0183] Early stopping strategy: If the performance of the validation set does not improve within a certain number of epochs, stop training early to prevent overfitting.

[0184] The model using the above training steps and strategies achieved excellent performance on multiple benchmark datasets, as shown in Table 1.

[0185] Table 1 Performance comparison of different methods

[0186]

[0187] The following are the experimental results of some key data sets:

[0188] MSVR310 dataset: The proposed model achieved a mAP of 40.0% and a rank1 accuracy of 50.4% on this dataset.

[0189] RGBNT201 dataset: The proposed model achieved 72.5% mAP and 78.3% rank1 accuracy on this dataset.

[0190] RGBNT100 dataset: The proposed model achieved 83.2% mAP and 97.2% rank1 accuracy on this dataset.

[0191] Market1501MM dataset: The proposed model achieved 80.4% mAP and 92.3% rank1 accuracy on this dataset.

[0192] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the invention.

Claims

1. A multimodal object re-identification method based on cue learning, characterized in that: The following steps are involved: S1: Construct a multimodal person re-identification model based on cue learning; S2: Establish a loss function to train the multimodal person re-identification model based on cue learning; S3: Apply the trained multimodal person re-identification model based on cue learning to actual scenarios to complete multimodal object re-identification based on cue learning; The multimodal person re-identification model based on cue learning in S1 includes: A modality feature extraction module, wherein the modality feature extraction module is used to extract global feature information from a multimodal image; A modality hint vector module, which is used to establish a communication channel between different modality features, and to capture and transfer modality complementary information; The modal feature fusion module is used to realize the interaction and fusion of different modal features; The modality missing scenario adaptation module is used to dynamically adjust the model structure; The modality feature extraction module includes a block encoder and a multi-layer serial self-attention feature extractor in sequence; The block encoder is represented as: in, is the visible light modality block coding sequence, is a visible light image, is a block encoder, is the visible light modal characteristic sequence, Encode the class, is the vector concatenation operation, Encode for position; in, is the near infrared modal block coding sequence, is a near-infrared image, is the near-infrared modal feature sequence; in, is the thermal infrared modality block encoding sequence, For thermal infrared images, is the thermal infrared modal feature sequence; The self-attention feature extractor includes a serial multi-head self-attention module, a feedforward neural network and a layer normalization module, which is expressed as: in, For the The visible light feature sequence output by the multi-head attention layer in the layer self-attention feature extractor, is the self-attention feature extractor The visible light image input of the layer, is a multi-head self-attention module, is a feed-forward neural network, is the layer normalization module; in, For the The near-infrared feature sequence output by the multi-head attention layer in the layer self-attention feature extractor, is the self-attention feature extractor Layer near infrared image input; in, For the The thermal infrared feature sequence output by the multi-head attention layer in the layer self-attention feature extractor, is the self-attention feature extractor Thermal infrared image input of the layer; The modal prompt vector module includes a modal prompt vector injection module and a prompt exchange module; The modality hint vector injection module introduces a modality hint vector into the input space of the modality feature extraction module, wherein the modality hint vector It is expressed as: in, is a learnable vector, is the dimension of the feature vector; The modal prompt vector Concatenated with the vector cube formed by stacking feature vectors in the block encoder , complete the injection of modal prompt vector; in, is the vector cube, Indicates the operation of stacking tensors. is the set of real numbers, is the length of the feature vector, is the visible light modal characteristic sequence, is the near-infrared modal feature sequence, is the thermal infrared modal feature sequence; The prompt exchange module is located at the output layer of the self-attention feature extractor and changes the position of the modal prompt vector through the transposition operation, which is expressed as: in, is the self-attention feature extractor The vector cube of the layer, is the self-attention feature extractor The modal hint vector of the layer, superscript Indicates that Transpose along the first and second dimensions; The modal feature fusion module includes a modal feature high response area selection module and a multimodal feature fusion module based on a modal prompt vector connected in series, and the modal feature high response area selection module is serially connected to a self-attention feature extractor in the modal feature extraction module; The modal feature high response region selection module selects high response feature vectors from two perspectives: inter-modal vector selection and intra-modal vector selection. The inter-modal vector selection is expressed as: in, is the modal key area mask, To generate a key region mask based on the element index, To find the top ranked element from a set of elements Elements of and are the attention scores associated with the modality cue, respectively; The modal internal vector selection is expressed as: in, is the intramodal key area mask, Attention scores for class encoding and image feature encoding; The joint mask selected by the modality feature high response region selection module It is expressed as: in, Indicates the union operation; The multimodal feature fusion module based on the modal cue vector uses the modal cue vector as the query vector and guides the fusion of features of different modalities through a cross-attention mechanism. The formula of the cross-attention mechanism is: in, is the cross attention layer, For the layer hint vector, For , and In the example, the high-response local feature vector sequence selected by the modal feature high-response region selection module is is the normalized exponential function, , and are the mapping matrices of query, keyword and value respectively, is the tensor dot product operator; The fusion layer of the multimodal feature fusion module based on modal cue vector is expressed as: in, is the intermediate variable, is a multi-layer perceptron, is the final modal fusion feature, is the feature sequence output by the feature fusion module of the last layer, It is the layer normalization module.

2. The multimodal target re-identification method based on prompt learning according to claim 1, characterized in that: The modality missing scene adaptation module includes a prompt vector shielding module and a feature sequence adjustment module, wherein the prompt vector shielding module and the feature sequence adjustment module are located between the block encoder and the first layer self-attention feature extractor; The prompt vector masking module adjusts the structure and operation of the model through prompts related to the missing modality when the modality is detected to be missing; The feature sequence adjustment module adjusts the multimodal token sequence in the case of modality loss.

3. The multimodal target re-identification method based on prompt learning according to claim 1, characterized in that: The loss function in S2 Combining triplet loss and cross entropy loss , the formula is: in, is the feature distance between the anchor sample and the positive sample, is the feature distance between the anchor sample and the negative sample, is a soft margin hyperparameter, For Category The smoothing label of The category predicted by the model The probability of is the number of categories.

Images