Multi-modal multi-task target re-identification method and system based on unbiased prototype consistency learning

Abstract

The invention discloses a multi-modal multi-task target re-identification method based on unbiased prototype consistency learning, and the method comprises the steps: firstly, employing a pre-trained visual model as a shared encoder, and extracting depth features from to-be-processed images of different modals; secondly, statistical distribution differences caused by different sensors are eliminated through a modal normalization layer, an unbiased memory bank and a consistency learning strategy are introduced, and modal deviation is corrected in a prototype learning mode; then, a cross-modal alignment mechanism is utilized to close the feature distance of the same kind of targets in different modals, and a hierarchical clustering algorithm is combined to automatically mine the semantic hierarchical structure between the targets, so that cross-category knowledge sharing and discriminative constraint are realized; and finally, constructing a multi-task joint loss function integrating identity recognition, clustering optimization, unbiased learning and hierarchical consistency to carry out end-to-end training, thereby realizing efficient and accurate recognition of various targets such as pedestrians and vehicles in a cross-modal scene under a unified framework.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, deep learning and multimodal intelligent recognition technology, and more specifically, relates to a multimodal multi-task target re-identification method and system based on unbiased prototype consistency learning. Background Technology

[0002] With the rapid development of artificial intelligence and multimodal learning technologies, object re-identification (ReID) has become a key task in fields such as intelligent security, autonomous driving, and video retrieval. The core of object re-identification lies in accurately identifying and matching specific targets (such as pedestrians and vehicles) across different camera fields of view, time periods, or imaging sensors. In current urban surveillance and complex traffic scenarios, massive amounts of video data contain rich spatiotemporal information. How to fully utilize this multimodal data to achieve efficient target retrieval and tracking is of paramount practical significance for maintaining public safety and improving the level of smart city governance. Therefore, developing a robust and efficient object re-identification framework has become a core problem urgently needing to be solved in the field of computer vision.

[0003] Currently, mainstream target re-identification methods can be categorized into the following types: The first is modal feature extraction based on independent branches, such as RGB-IR re-identification models. These methods typically construct separate deep learning network branches for visible and infrared light, aiming to extract discriminative features within each modality. The second is feature alignment methods based on a unified embedding space. These methods map images from different modalities to the same global semantic space through contrastive or metric learning, achieving cross-modal matching by narrowing the feature distance between similar targets in heterogeneous modalities. The third is multi-task learning methods based on specific category divisions, where specialized re-identification models are trained separately for different target categories such as pedestrians and vehicles. By optimizing the loss function within their respective task spaces, the model's accuracy in recognizing specific target categories is improved.

[0004] However, the aforementioned existing target re-identification methods all have some significant drawbacks in practical applications: First, the aforementioned methods based on independent branches are prone to modal bias and inconsistency with the task. Because the feature extraction processes of different modalities are relatively independent, the feature distribution of the model is often extremely uneven in complex scenarios across modalities and tasks, resulting in poor cross-domain generalization ability and difficulty in adapting to changing real-world environments. Second, the above-mentioned methods based on unified embedding space ignore the semantic differences between modalities, which can easily lead to modal degradation (for example, visible light images focus on detailed textures, while infrared images focus on thermal intensity distribution). Traditional forced alignment methods will over-compress the feature space, resulting in the loss of key discriminative information unique to each modality, thereby reducing the accuracy of re-identification. Third, the aforementioned multi-task learning methods cannot achieve full sharing of modality-independent knowledge. They typically treat the re-identification of different categories as isolated tasks, ignoring the common features that different target categories (such as people and vehicles) may have in a multimodal environment, resulting in low model learning efficiency.

[0005] Fourth, the three existing methods mentioned above have high costs in the training and inference stages. Due to the lack of a unified optimization framework, existing cross-modal or multi-task models are often bloated and require repeated training of multiple models for different scenarios. This makes it difficult to achieve efficient joint recognition across modalities and tasks and fails to meet the application requirements of large-scale real-time systems. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a multimodal, multi-task target re-identification method and system based on unbiased prototype consistency learning. Its purpose is to solve the technical problems of existing methods based on independent branches, which easily lead to modal bias and task inconsistency, and poor cross-domain generalization; existing methods based on unified embedding spaces, which ignore modal semantic differences, easily causing modal degradation and loss of key information; existing multi-task learning methods, which ignore the common features that different target categories may have in a multimodal environment, resulting in low model learning efficiency; and the technical problems of the three existing methods, which have bloated model structures, require independent training, and have high training and inference costs.

[0007] To achieve the above objectives, according to one aspect of the present invention, a multimodal, multi-task target re-identification method based on unbiased prototype consistency learning is provided, comprising the following steps: (1) Acquire the image to be recognized; (2) Perform data preprocessing on the image to be identified obtained in step (1) to obtain the preprocessed image; (3) Input the preprocessed image obtained in step (2) into the pre-trained multimodal unbiased recognition model to obtain the feature representation of the image; (4) Compare the feature representation of the image obtained in step (3) with the features of multiple samples registered in the pre-established target library to obtain multiple similarity matching results, and sort all similarity matching results in descending order; (5) Obtain the sample feature corresponding to the highest-ranked similarity matching result in the target library from the similarity matching results obtained in step (4), and use the sample and its confidence score corresponding to the sample feature in the target library as the final recognition result.

[0008] Preferably, step (2) specifically involves first adjusting the size of the image to be recognized to 256×128×3, where 256×128 represents the resolution of the image to be recognized and 3 represents the number of RGB channels; then, performing one or any combination of image normalization, random flipping, and random cropping on the resized image to obtain the preprocessed image. The multimodal unbiased recognition model includes an image patch embedding layer, a Transformer feature encoding layer, an RGB modality normalization layer, an NI modality normalization layer, a TI modality normalization layer, an identity classification layer, a clustering memory layer, an unbiased feature fusion layer, a cross-modal alignment layer, a hierarchical clustering constraint layer, and a multi-task loss optimization layer. The input to the image patch embedding layer consists of three modalities: a first visible light RGB image, an NI image, and a TI image. Each image has dimensions of 256×128×3. The image patch embedding layer performs the following operations on each modal image: First, it performs convolution operations on the first visible light RGB image, NI image, and TI image respectively to divide them into 128 image patches of 16×8 dimensions, thereby obtaining feature maps corresponding to the visible light RGB image, NI image, and TI image with dimensions of 16×8×768. Then, it flattens the feature maps corresponding to the first visible light RGB image, NI image, and TI image from the spatial dimension into a first visible light RGB image of length 128. The feature sequences corresponding to the first visible light RGB image, NI image, and TI image have a feature dimension of 768 at each sequence position. Next, a learnable class label with a dimension of 1×768 is inserted at the beginning of the feature sequences corresponding to the first visible light RGB image, NI image, and TI image, expanding the length of the feature sequences corresponding to the first visible light RGB image, NI image, and TI image to 129 respectively. Subsequently, the learnable positional encoding is added element-wise to the feature sequences corresponding to the first visible light RGB image, NI image, and TI image respectively. Finally, layer normalization is performed on the summed feature sequences to obtain the embedded feature sequences corresponding to the first visible light RGB image, NI image, and TI image respectively, and then output.

[0009] Preferably, the input to the Transformer feature coding layer is the embedding feature sequences corresponding to the first visible light RGB image, NI image, and TI image output by the image block embedding layer. The Transformer feature coding layer inputs the embedding feature sequences corresponding to the first visible light RGB image, NI image, and TI image to an encoder containing 12 Transformer coding blocks for deep feature extraction, so as to output 12 coding feature sequences with dimensions of 129×512 corresponding to the first visible light RGB image, 12 coding feature sequences with dimensions of 129×512 corresponding to the NI image, and 12 coding feature sequences with dimensions of 129×512 corresponding to the TI image. All Transformer coding blocks have the same structure, and each Transformer coding block performs the following operations for each embedded feature sequence F: (a1) Perform layer normalization on the embedded feature sequence F to obtain the normalized feature sequence F1; (a2) Perform a multi-head self-attention operation on the normalized feature sequence F1 obtained in step (a1) to obtain a feature sequence F2 with dimension L×D; Specifically, this step involves the following steps: First, three linear transformations are used to generate a query matrix Q, a key matrix K, and a value matrix V, each with a dimension of 129×512. Then, the query matrix Q, the key matrix K, and the value matrix V are divided into 12 heads according to their feature dimensions, each head having a dimension of 129×64. Next, the attention weight Softmax((Q·K^T) / √64) for each head is obtained. Subsequently, this attention weight is multiplied by the value matrix V to obtain a weighted feature. Finally, the weighted features of the 12 heads are concatenated and then linearly projected to obtain a feature sequence F2 with a dimension of L×D. (a3) Add the feature sequence F2 obtained in step (a2) to the embedded feature sequence F element by element to obtain the feature sequence F3; (a4) Perform layer normalization on the feature sequence F3 obtained in step (a3) ​​to obtain the normalized feature sequence F4; (a5) Perform feedforward neural network transformation on the normalized feature sequence F4 obtained in step (a4). First, expand the dimension of the normalized feature sequence F4 from 768 to 3072 through a linear layer. Then, activate the normalized feature sequence after dimension expansion using the GELU activation function. Finally, compress the dimension of the normalized feature sequence after activation back from 3072 to 768 through a linear layer to obtain the feature sequence F5. (a6) Add the feature sequence F5 obtained in step (a5) to the feature sequence F3 obtained in step (a3) ​​element by element to obtain an L×D encoded feature sequence, which is used as the output of the Transformer encoded block.

[0010] Preferably, the input to the RGB modality normalization layer is the encoded feature sequence corresponding to the first visible light RGB image output by the Transformer feature encoding layer. The RGB modality normalization layer first performs batch normalization on the encoded feature sequence; then, it performs batch mean and variance processing on the batch normalization calculation results to obtain the mean μ and variance σ²; subsequently, it normalizes the obtained mean μ and variance σ² using the formula (x-μ) / √(σ²+ε), where x represents the global feature vector at position 0 in the encoded feature sequence to obtain the normalization result F; finally, it performs an affine transformation on the obtained normalization result F with a learnable scaling parameter γ to obtain a normalized feature vector corresponding to the visible light RGB image with a dimension of 512. The input to the NI modality normalization layer is the encoded feature sequence corresponding to the NI image output by the Transformer feature coding layer. This layer first performs the same processing on the encoded feature sequence as the RGB modality normalization layer mentioned above to obtain a normalized feature vector corresponding to the NI image with a dimension of 512. The input to the TI modality normalization layer is the encoded feature sequence corresponding to the TI image output by the Transformer feature encoding layer. This layer performs the same processing on the encoded feature sequence as the RGB modality normalization layer above to obtain a normalized feature vector corresponding to the TI image with a dimension of 512. The input to the identity classification layer consists of the normalized feature vectors corresponding to the first visible light RGB image output by the RGB modality normalization layer, the normalized feature vectors corresponding to the NI image output by the NI modality normalization layer, the normalized feature vectors corresponding to the TI image output by the TI modality normalization layer, the normalized feature vectors obtained from the second visible light RGB image after processing by the image patch embedding layer, the Transformer feature encoding layer, and the RGB modality normalization layer, and the identity label vectors corresponding to the aforementioned modalities. This layer performs linear classification and loss calculation on each input normalized feature vector to obtain the classification score corresponding to the first visible light RGB image. The classification score corresponding to the second visible light RGB image. Classification scores corresponding to NI images Classification score corresponding to TI image The normalized feature vector corresponding to the first visible light RGB image The normalized feature vector corresponding to the second visible light RGB image. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images and multimodal identity recognition loss ; The processing steps of the identity classification layer include the following: (b1) Perform a linear transformation on the normalized feature vector corresponding to the first visible light RGB image to obtain the classification score corresponding to the first visible light RGB image. Its dimension is C; (b2) The classification score corresponding to the first visible light RGB image obtained in step (b1). The identity tag corresponding to the visible light RGB image is used to obtain the RGB identity loss corresponding to the first visible light RGB image. ; Specifically, this step involves obtaining the RGB identity loss. The following formula is used: ; in The model predicts that the sample belongs to the first... The probability of a class is determined by its classification score. The Middle Each component Calculated using the Softmax function, i.e. ; Let be the probability distribution of the i-th true label after label smoothing. When the category corresponds to the real identity label of the sample, ,otherwise ,in This is the label smoothing coefficient, with a value of 0.1; (b3) The normalized feature vector corresponding to the first visible light RGB image The RGB triplet loss corresponding to the first visible light RGB image is obtained by matching the identity tag with the first visible light RGB image. ; Specifically, this step involves obtaining the RGB triplet loss corresponding to the first visible light RGB image. The following formula is used:

[0011] in This represents the maximum Euclidean distance between the normalized feature vector of the first visible light RGB image and all positive samples in the same batch of training samples; where a positive sample is a sample with the same identity as the first visible light RGB image, and the maximum Euclidean distance is the Euclidean distance between the normalized feature vector of the sample that is furthest away from all positive samples. This represents the minimum Euclidean distance between the normalized feature vector of the first visible light RGB image and all negative samples in the same batch of training samples; where a negative sample is a sample with a different identity from the first visible light RGB image, and the minimum Euclidean distance is the Euclidean distance between the normalized feature vector of the closest negative sample and the first negative sample; the final RGB triplet loss. This is the average loss of all triplet samples within the same batch; (b4) Loss the RGB identity corresponding to the first visible light RGB image obtained in step (b2). The triplet loss corresponding to the first visible light RGB image obtained in step (b3) A weighted combination is performed to obtain the modal loss corresponding to the first visible light RGB image. ; (b5) Obtain the second visible light RGB image and its corresponding modal loss. And obtain the modal loss corresponding to the NI image. and the modal loss corresponding to the TI image. ; (b6) The modal loss corresponding to the visible light RGB image obtained in step (b4) and the modal loss corresponding to the second visible light RGB image obtained in step (b5). Modal loss corresponding to NI image and the modal loss corresponding to the TI image. Add them together to obtain the multimodal identity recognition loss. It outputs the classification score corresponding to the first visible light RGB image. The classification score corresponding to the second visible light RGB image. Classification scores corresponding to NI images Classification score corresponding to TI image The normalized feature vector corresponding to the first visible light RGB image The normalized feature vector corresponding to the second visible light RGB image. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images and multimodal identity recognition loss .

[0012] Preferably, the input to the clustering memory layer is the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. and the normalized feature vectors corresponding to the identity tag vectors and NI images. and the normalized feature vectors corresponding to the identity label vectors and TI images. And identity tag vectors, as well as the RGB image memory updated in the last iteration. NI Image Memory and TI Image Memory This layer performs a clustering memory operation on all inputs to obtain the clustering constraint loss. and the updated RGB image memory library NI Image Memory and TI Image Memory And output; The processing of the clustering memory layer includes the following steps: (c1) The normalized feature vector corresponding to the first visible light RGB image L2 normalization is performed to obtain the unit feature vector corresponding to the first visible light RGB image. ; (c2) Calculate the unit eigenvector The RGB image memory updated in the previous iteration The inner product similarity of each category prototype is calculated, and the dimensions of all inner product similarities are denoted by . Similarity vector ; (c3) Transfer the similarity vector Divide by temperature coefficient To obtain the scaled logits vector ; (c4) For the scaled logits vector Cross-entropy loss is calculated using the identity label vector corresponding to the first visible light RGB image to obtain the clustering loss corresponding to the first visible light RGB image. ; (c5) Update the RGB image memory after the previous iteration Update to obtain the RGB image memory library updated in this iteration. For each sample in the current batch, based on its identity tag Update its corresponding identity prototype: And then to The updated identity prototype in Normalization processing is performed to obtain the normalized RGB image memory updated in this iteration. ; (c6) Following the same method as steps (c1) to (c5) above, normalize the feature vector corresponding to the NI image. Processing is performed to obtain the clustering loss corresponding to the NI image. And the NI image memory updated in this iteration And the normalized feature vector corresponding to the TI image. Processing is performed to obtain the clustering loss corresponding to the TI image. And the TI image memory updated in this iteration ; (c7) The clustering loss corresponding to the first visible light RGB image obtained NI image corresponding clustering loss The clustering constraint loss is obtained by adding the clustering loss corresponding to the TI image and the clustering loss corresponding to the TI image. .

[0013] Preferably, the input to the unbiased feature fusion layer is the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. and the normalized feature vectors corresponding to the identity tag vectors and NI images. and the normalized feature vectors corresponding to the identity label vectors and TI images. Together with the identity label vector and the unbiased fusion feature vector updated in the previous iteration. Its initial value is a matrix of all zeros with dimension . This layer performs a clustering memory operation on all inputs to obtain an unbiased learning loss. And the unbiased memory updated in this round of iterations And output; The processing steps of the unbiased feature fusion layer include the following: (d1) Normalized feature vector corresponding to the visible light RGB image Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The formula is to add elements one by one and take the average. To obtain unbiased fusion feature vectors; (d2) The unbiased fusion feature vector obtained in step (d1) Perform L2 normalization to obtain unit eigenvectors. ; (d3) Calculate the unit eigenvector obtained in step (d2). The unbiased clustering memory updated in the previous iteration The inner product similarity of each identity prototype, and the dimensions of all inner product similarities are . Similarity vector ; (d4) The similarity vector obtained in step (d3) Divide by temperature coefficient To obtain the scaled logits vector ; (d5) The scaled logits vector obtained in step (d4) Cross-entropy loss is calculated using the corresponding identity label vectors to obtain the unbiased clustering loss. ; Specifically, this step involves first processing the scaled logits vector. Apply the Softmax function to obtain the predicted probability for each identity. Then, based on the sample's real identity label Calculate the corresponding negative logarithmic probability as the loss value: ; (d6) Apply the unbiased clustering loss obtained in step (d5) Multiply by unbiased weighting coefficient To obtain the weighted unbiased learning loss ; (d7) Use the unit eigenvector obtained in step (d2) The unbiased memory updated in the previous iteration Update to obtain the unbiased memory after this iteration. .

[0014] Preferably, the input to the cross-modal alignment layer is the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images This layer performs cross-modal contrastive learning on all inputs to obtain a weighted cross-modal contrastive loss. And output; The process of processing cross-modal alignment layers includes the following steps: (e1) Based on the normalized feature vector corresponding to the first visible light RGB image Normalized feature vectors corresponding to NI images and the normalized feature vector corresponding to the TI image. Obtain the contrastive loss between pairwise normalized feature vectors; Specifically, the normalized feature vector corresponding to the first visible light RGB image is calculated. Normalized feature vectors corresponding to NI images The specific comparison loss includes: (e1a) The normalized feature vector corresponding to the first visible light RGB image Normalized feature vectors corresponding to NI images L2 normalization was performed separately to obtain the unit eigenvalue matrix. and ; (e1b) The unit characteristic matrix obtained in calculation step (e1a) With the unit characteristic matrix The matrix product of the transposes of the two matrices is used as the similarity matrix between them. ; (e1c) The similarity matrix obtained in step (e1b) Divide by temperature coefficient To obtain the logits matrix; (e1d) Construct the label vector , indicating that the diagonal position is a positive sample pair, where This indicates the batch size, which is the number of samples in the current training batch; (e1e) is the logits matrix obtained in step (e1c). The label vector obtained from each row and step (e1d) For each value in the table, cross-entropy loss is calculated to obtain the loss in the RGB to NI direction. ; (e1f) The logits matrix obtained in step (e1c) Each row of the transpose matrix and the tag vector obtained in step (e1d) Perform cross-entropy loss calculation to obtain the loss from the NI to the RGB direction. ; (e1g) Take the loss from RGB to NI direction obtained in step (e1e). The loss from NI to RGB direction obtained in step (e1f) The average value is used to obtain the normalized feature vector corresponding to the first visible light RGB image. Normalized feature vectors corresponding to NI images Comparison loss between ; (e2) Using the method described in step (e1) above, calculate the normalized feature vector corresponding to the first visible light RGB image. Normalized feature vectors corresponding to TI images Cross-modal contrast loss And the normalized feature vector corresponding to the NI image. Normalized feature vectors corresponding to TI images Cross-modal contrast loss ; (e3) The normalized feature vector corresponding to the first visible light RGB image obtained in step (e1) Normalized feature vectors corresponding to NI images Comparison loss between and the normalized feature vector corresponding to the first visible light RGB image obtained in step (e2). Normalized feature vectors corresponding to TI images Cross-modal contrast loss and the normalized feature vector corresponding to the NI image. Normalized feature vectors corresponding to TI images Cross-modal contrast loss Add to obtain the loss ; (e4) The loss obtained in step (e3) Multiply by cross-modal weighting coefficient To obtain weighted cross-modal contrast loss And output it.

[0015] Preferably, the input to the hierarchical clustering constraint layer is the updated unbiased memory output by the unbiased feature fusion layer. Human identity ID set and vehicle identity ID set And the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. Normalized feature vectors corresponding to NI images and the normalized feature vector corresponding to the TI image. and the target category label vector corresponding to the current batch of samples. This layer contains a set of human identity IDs. and vehicle identity ID set Perform the first integer neighbor clustering operation to obtain the hierarchical semantic constraint loss. And output; The process of hierarchical clustering constraint layer includes the following steps: (f1) from unbiased memory Extract the prototype feature vector matrix for the human category. Its dimensions are ; (f2) The prototype eigenvector matrix obtained in step (f1) Perform FINCH hierarchical clustering (parameters: cosine distance as the distance metric, no premature termination, no output) to obtain the hierarchical clustering result matrix. and the number of clusters at each level ; (f3) The hierarchical clustering result matrix obtained in step (f2) Traverse all clusters at each level and obtain the mean of all prototype feature vectors in each cluster as the central feature vector of that cluster. The central feature vectors of all clusters in that level constitute the set of cluster central feature vectors for that level. (f4) Calculate the average of the cluster center feature vector sets obtained in step (f3) for all levels to obtain the human superclass prototype feature vector. Its dimension is 512; (f5) From unbiased memory The prototype feature vector matrix of the vehicle category is obtained. Following the same parameters and methods as steps (f1) to (f4), FINCH hierarchical clustering is performed on the prototype feature vector matrix of the vehicle category to obtain the prototype feature vector of the vehicle superclass. Its dimension is 512; (f6) The human superclass prototype feature vector obtained in step (f4) The vehicle superclass prototype feature vector obtained in step (f5) Concatenate rows to obtain the superclass prototype matrix. Its dimensions are ; (f7) Based on the target category label vector of the current batch of samples The superclass prototype matrix obtained from step (f6) Obtain the target category label vector. Corresponding batch prototype matrix Its dimensions are ; (f8) Calculate the normalized feature vectors corresponding to the first visible light RGB image respectively. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The batch prototype matrix obtained in step (f7) Mean square error loss between , ,as well as : in: ; ; ; in This represents the superclass prototype corresponding to each sample in the current batch; (f9) The mean squared error loss obtained in step (f8) , ,as well as Add them together to get the total loss. ; (f10) The total loss obtained in step (f9) Multiply by the hierarchical clustering weight coefficient To obtain the weighted hierarchical semantic constraint loss ; The input to the multi-task loss optimization layer is the multimodal identity recognition loss output by the identity classification layer. Clustering constraint loss output by the clustering memory layer Unbiased learning loss output by the unbiased feature fusion layer Cross-modal contrast loss output of cross-modal alignment layer And the hierarchical clustering loss output by the hierarchical clustering constraint layer. This layer sums all the losses to obtain the total loss. Then, a mixed-precision training strategy is used to adjust the total loss. Perform backpropagation optimization to obtain the parameters of the optimized multimodal unbiased recognition model for the next batch of training: The processing steps of the multi-task loss optimization layer include the following: (g1) The multimodal identity recognition loss output by the identity classification layer Clustering constraint loss output by the clustering memory layer Unbiased learning loss output by the unbiased feature fusion layer Cross-modal contrast loss output of cross-modal alignment layer And the hierarchical clustering loss output by the hierarchical clustering constraint layer. Add them together to get the total loss. And use a gradient scaler to adjust the total loss Perform dynamic scaling to capture the loss after scaling. ; (g2) The scaled loss obtained in step (g1) Perform backpropagation optimization to obtain the scaling gradients of all learnable parameters of the multimodal unbiased recognition model; (g3) Determine whether the scaling gradient obtained in step (g2) overflows. If overflow occurs, the process ends. Otherwise, restore the scaling gradient to the true gradient. Based on the true gradient, use the Adam optimizer to update the parameters of the multimodal unbiased recognition model to obtain the optimized parameters of the multimodal unbiased recognition model. The specific parameters of the Adam optimizer are: base learning rate. The exponential decay rate estimated by the first moment The exponential decay rate estimated by the second moment The weight decay coefficient is 0.0005.

[0016] Preferably, the multimodal unbiased recognition model is trained through the following steps: (3-1) Obtain the RGBNT201 and RGBNT100 datasets as training sets, and obtain the MSVR310 dataset as a test set; (3-2) Perform data preprocessing on the training set obtained in step (3-1) to obtain the preprocessed training set; (3-3) For each sample in the preprocessed training set obtained in step (3-2), the first visible light RGB image in the sample is input into the first layer of the multimodal unbiased recognition model for image patch embedding processing, so as to output the embedded feature sequence corresponding to the first visible light RGB image of the sample, with a dimension of 129×768. Each sample contains two visible light RGB images, NI image and TI image, and each image has a dimension of 256×128×3. (3-4) For each sample in the preprocessed training set obtained in step (3-2), the embedded feature sequence corresponding to the first visible light RGB image of the sample obtained in step (3-3) is input into the second layer of the multimodal unbiased recognition model for Transformer feature encoding to output the global feature vector corresponding to the first visible light RGB image of the sample. (3-5) For each sample in the preprocessed training set obtained in step (3-2), the global feature vector corresponding to the first visible light RGB image of the sample obtained in step (3-4) is input into the third layer of the multimodal unbiased recognition model for batch normalization processing, so as to output the normalized feature vector corresponding to the first visible light RGB image of the sample. The dimension is 512; (3-6) For each sample in the preprocessed training set obtained in step (3-2), the NI image of the sample is successively input into the first, second, and fourth layers of the multimodal unbiased recognition model for processing, so as to output the normalized feature vector corresponding to the NI image of the sample. The dimension is 512; (3-7) For each sample in the preprocessed training set obtained in step (3-2), the TI image of the sample is sequentially input into the first, second, and fifth layers of the multimodal unbiased recognition model for processing, so as to output the normalized feature vector corresponding to the TI image of the sample. The dimension is 512; (3-8) For each sample in the preprocessed training set obtained in step (3-2), the second visible light RGB image corresponding to the sample is sequentially input into the first, second, and third layers of the multimodal unbiased recognition model for processing, so as to output the normalized feature vector corresponding to the second visible light RGB image of the sample. The dimension is 512; (3-9) For each sample in the preprocessed training set obtained in step (3-2), the four modality-normalized feature vectors corresponding to that sample obtained in steps (3-5), (3-6), (3-7), and (3-8) are... , , , The corresponding identity label is input into the 6th layer of the multimodal unbiased recognition model for identity classification processing to obtain the classification score corresponding to the first visible light RGB image of the sample. The classification score corresponding to the second visible light RGB image. Classification scores corresponding to NI images Classification score corresponding to TI image and multimodal identity recognition loss ; (3-10) For each sample in the preprocessed training set obtained in step (3-2), the normalized feature vector corresponding to the first visible light RGB image of that sample obtained in steps (3-5), (3-6), and (3-7) is used. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The corresponding identity label is input into the 7th layer of the multimodal unbiased recognition model for clustering memory processing to obtain the clustering constraint loss corresponding to the sample. ; (3-11) For each sample in the preprocessed training set obtained in step (3-2), the normalized feature vector corresponding to the first visible light RGB image of that sample obtained in steps (3-5), (3-6), and (3-7) is... Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The corresponding identity label is input into the 8th layer of the multimodal unbiased recognition model for unbiased feature fusion processing to obtain the unbiased learning loss corresponding to the sample. and unbiased memory bank ; (3-12) For each sample in the preprocessed training set obtained in step (3-2), the normalized feature vector corresponding to the first visible light RGB image of that sample obtained in steps (3-5), (3-6), and (3-7) is used. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The 9th layer of the input multimodal unbiased recognition model is subjected to cross-modal alignment processing to obtain the cross-modal contrast loss corresponding to the sample. ; (3-13) For each sample in the preprocessed training set obtained in step (3-2), the unbiased memory corresponding to that sample obtained in step (3-11) is... Human identity ID set and vehicle identity ID set And the three modality-normalized feature vectors corresponding to the sample obtained in steps (3-5), (3-6), and (3-7). , , The first integer neighbor clustering operation is performed on the 10th layer of the multimodal unbiased recognition model to obtain the hierarchical semantic constraint loss corresponding to the sample. ; (3-14) For each sample in the preprocessed training set obtained in step (3-2), the multimodal identity recognition loss corresponding to that sample is obtained in step (3-9). The clustering constraint loss corresponding to the sample obtained in step (3-10) The unbiased learning loss corresponding to the sample obtained in step (3-11) The cross-modal contrast loss corresponding to the sample obtained in step (3-12) The hierarchical semantic constraint loss corresponding to the sample obtained in step (3-13) Calculate the total loss corresponding to this sample. ; (3-15) For each sample in the preprocessed training set obtained in step (3-2), based on the total loss corresponding to the sample obtained in step (3-14), a mixed precision training strategy is used to perform backpropagation and parameter update operations on the multimodal unbiased recognition model. (3-16) For the preprocessed training set obtained in step (3-2), repeat steps (3-3) to (3-15) until the multimodal unbiased recognition model reaches the preset number of training cycles, and obtain the optimal parameters of the multimodal unbiased recognition model at this time, so as to obtain the initially trained multimodal unbiased recognition model. (3-17) Use the test set obtained in step (3-1) to test and verify the multimodal unbiased recognition model initially trained in step (3-16) until the recognition accuracy reaches the optimal level, thereby obtaining the final trained multimodal unbiased recognition model.

[0017] According to another aspect of the present invention, a multimodal, multi-task target re-identification system based on unbiased prototype consistency learning is provided, comprising the following modules: The first module is used to acquire the image to be recognized; The second module is used to preprocess the image to be identified obtained by the first module to obtain a preprocessed image. The third module is used to input the preprocessed image obtained from the second module into the pre-trained multimodal unbiased recognition model to obtain the feature representation of the image. The fourth module is used to compare the feature representation of the image obtained by the third module with the features of multiple samples registered in the pre-established target library to obtain multiple similarity matching results, and sort all similarity matching results in descending order; The fifth module is used to obtain the sample features corresponding to the highest-ranked similarity match result obtained from the fourth module in the target database, and uses the sample features and their corresponding samples and confidence scores in the target database as the final recognition result. In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. The present invention adopts the multi-task joint optimization strategy and unbiased feature fusion layer in step (3). By averaging the normalized feature vectors of different modes and combining them with the unbiased memory for constraint, it can learn cross-modal consistent feature representations and eliminate modality-specific biases. Therefore, it can solve the technical problems of existing independent branch-based methods that easily lead to modality bias and inconsistency with tasks, and poor cross-domain generalization. 2. This invention employs a modality normalization layer and a cross-modality alignment layer. By setting independent batch normalization processing for each modality to eliminate statistical distribution differences, and by using cross-modal contrastive learning to narrow the distance between different modal features of the same identity, it can retain the discriminative information of each modality and establish accurate modality correspondence. Therefore, it can solve the technical problems of existing methods based on a unified embedding space ignoring modal semantic differences and easily causing modality degradation and loss of key information. 3. This invention employs a hierarchical clustering constraint layer, which automatically discovers the semantic hierarchical structure between categories through the FINCH hierarchical clustering algorithm and aggregates the categories into supercategories for unified constraint. This introduces a higher-level semantic supervision signal, prompting each modal feature to follow the same semantic hierarchical structure. Therefore, it can solve the technical problem of low model learning efficiency caused by existing multi-task learning methods ignoring the common features that different target categories may have in a multimodal environment. 4. This invention adopts a multi-task loss optimization layer and a shared visual encoder structure, which integrates five optimization objectives—identity recognition, clustering constraints, unbiased learning, cross-modal alignment, and hierarchical semantic constraints—and utilizes a hybrid precision training strategy to achieve efficient joint optimization and representation across modalities and tasks. Therefore, it can solve the technical problems of the above three existing methods, such as bloated model structure, need for independent training, and high training and inference costs. 5. This invention enhances the model's learning ability for difficult samples by introducing a clustering memory layer and utilizing dynamically updated category feature vector prototypes and temperature coefficient control, making the feature representation more compact and discriminative. 6. This invention employs a Transformer feature encoding layer based on a pre-trained CLIP model, which leverages the powerful visual representation capabilities obtained from large-scale image-text pair pre-training and captures global and local semantic dependencies through a multi-head self-attention mechanism, significantly improving the depth and accuracy of feature extraction. 7. The multi-task loss optimization layer of this invention adopts a hybrid precision training and dynamic gradient scaling strategy, which significantly reduces memory usage and shortens computation time while maintaining numerical stability, thus greatly improving the training efficiency of the model. 8. In the data preprocessing step, this invention adopts the P×K sampling strategy to ensure that each batch contains a sufficient number of positive and negative sample pairs, which provides a solid foundation for the calculation of triplet loss and clustering loss and further enhances the discriminative ability of the model. Attached Figure Description

[0018] Figure 1 This is a flowchart of the multimodal, multi-task target re-identification method based on unbiased prototype consistency learning according to the present invention; Figure 2 This is a schematic diagram of the structure of the multimodal unbiased recognition model of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0020] The basic idea of ​​this invention is to address the problems of modal bias, task isolation, and feature degradation in existing target re-identification methods. This invention proposes a multimodal, multi-task target re-identification framework based on unbiased prototype consistency learning. First, a pre-trained visual model is used as a shared encoder to extract deep features from images of different modalities. Second, a modality normalization layer eliminates statistical distribution differences caused by different sensors, and an unbiased memory and consistency learning strategy are introduced to correct modal bias through prototype learning, ensuring consistency of the feature space in multimodal, multi-task environments. Next, a cross-modal alignment mechanism is used to shorten the feature distance between similar targets in different modalities, and a hierarchical clustering algorithm is combined to automatically mine the semantic hierarchical structure between targets, achieving cross-category knowledge sharing and discriminative constraints. Finally, an end-to-end training is performed by constructing a multi-task joint loss function integrating identity recognition, clustering optimization, unbiased learning, and hierarchical consistency, thereby achieving efficient and accurate identification of multiple targets such as pedestrians and vehicles in cross-modal scenarios within a unified framework.

[0021] like Figure 1 As shown, this invention provides a multimodal, multi-task target re-identification method based on unbiased prototype consistency learning, comprising the following steps: (1) Acquire the image to be recognized; Specifically, this step involves acquiring images to be identified from image acquisition devices in visible light (RGB), near-infrared (NI), or thermal infrared (TI) modes, including target samples of different categories such as pedestrians and vehicles.

[0022] (2) Perform data preprocessing on the image to be identified obtained in step (1) to obtain the preprocessed image; Specifically, this step involves first adjusting the size of the image to be recognized to 256×128×3 (where 256×128 represents the resolution of the image and 3 represents the number of RGB channels); then, performing one or any combination of image normalization, random flipping, and random cropping on the resized image to obtain the preprocessed image.

[0023] The advantage of this step is that by using uniform image size adjustment and preprocessing operations, it eliminates image differences caused by different acquisition devices and environments, provides standardized input for subsequent feature extraction, and improves the stability and robustness of the model.

[0024] (3) Input the preprocessed image obtained in step (2) into the pre-trained multimodal unbiased recognition model to obtain the feature representation of the image; The multimodal unbiased recognition model of the present invention is a feature extraction model trained under cross-modal alignment constraints, unbiased learning constraints and hierarchical semantic constraints. It can map inputs from different modalities to a unified semantic space and extract discriminative feature representations.

[0025] The advantage of this step is that it achieves a unified representation of cross-modal inputs through a shared visual encoder and a multi-task joint optimization strategy, eliminating the feature shift problem caused by modal differences and providing a consistent and stable feature foundation for subsequent recognition.

[0026] (4) Compare the feature representation of the image obtained in step (3) with the features of multiple samples registered in the pre-established target library to obtain multiple similarity matching results, and sort all similarity matching results in descending order; Specifically, in this similarity comparison step, the corresponding features are selected based on the combination of query modality and library modality (such as RGB querying NI library, NI querying TI library, multimodal fusion retrieval, etc.), and the results are sorted by calculating cosine similarity to output a list of candidate results.

[0027] (5) Obtain the sample feature corresponding to the highest-ranked similarity matching result in the target library from the similarity matching results obtained in step (4), and use the sample and its confidence score corresponding to the sample feature in the target library as the final recognition result.

[0028] Specifically, this step outputs the matched samples and their confidence scores based on the similarity ranking results, which serve as the final identification results; at the same time, it calculates and outputs performance metrics such as Rank-1 accuracy (first-place hit rate), Rank-5 accuracy (top five hit rate), Rank-10 accuracy (top ten hit rate), and mAP (mean precision).

[0029] The advantage of this step is that, through flexible modality combination strategies and standardized evaluation metrics, it can comprehensively reflect the performance of the model in different retrieval scenarios, providing reliable technical support for practical applications.

[0030] The advantages of steps (1) to (5) above are that, through the collaborative design of multimodal input, feature sharing, unbiased enhancement and unified similarity calculation, the present invention constructs a unified recognition process that can run stably under different modal conditions, which can significantly improve the accuracy, robustness and generalization performance of cross-modal re-recognition.

[0031] like Figure 2 As shown, the multimodal unbiased recognition model of the present invention includes an image patch embedding layer, a Transformer feature encoding layer, an RGB modality normalization layer, an NI modality normalization layer, a TI modality normalization layer, an identity classification layer, a clustering memory layer, an unbiased feature fusion layer, a cross-modal alignment layer, a hierarchical clustering constraint layer, and a multi-task loss optimization layer. The structure of each layer is as follows: The image patch embedding layer takes three modalities of images as input: a first visible light RGB image, an NI image, and a TI image. Each image has dimensions of 256×128×3. The image patch embedding layer performs the following operations on each modal image: First, it performs convolution operations on the first visible light RGB image, NI image, and TI image respectively (operation parameters: 3 input channels, 768 output channels, 16×16 kernel size, 16 stride, 0 padding, no bias term) to divide the first visible light RGB image, NI image, and TI image into 128 image patches of 16×8 each, thereby obtaining feature maps corresponding to the visible light RGB image, NI image, and TI image with dimensions of 16×8×768. Then, it flattens the feature maps corresponding to the first visible light RGB image, NI image, and TI image from the spatial dimension into a first visible light RGB image of length 128. The feature sequences corresponding to the first visible light RGB image, NI image, and TI image have a feature dimension of 768 for each sequence position. Next, a learnable class token with a dimension of 1×768 is inserted at the beginning of the feature sequences corresponding to the first visible light RGB image, NI image, and TI image, expanding the length of each feature sequence to 129. Then, the learnable positional encoding (with a dimension of 129×768) is added element-wise to the feature sequences corresponding to the first visible light RGB image, NI image, and TI image (to encode spatial positional information). Finally, layer normalization is performed on the summed feature sequences to obtain the embedded feature sequences (with a dimension of 129×768) corresponding to the first visible light RGB image, NI image, and TI image, and output them. The input to the Transformer feature coding layer is the embedding feature sequences corresponding to the first visible light RGB image, NI image, and TI image output by the image block embedding layer. The Transformer feature coding layer inputs the embedding feature sequences corresponding to the first visible light RGB image, NI image, and TI image to the encoder containing 12 Transformer coding blocks for deep feature extraction, so as to output 12 coding feature sequences with dimensions of 129×512 corresponding to the first visible light RGB image, 12 coding feature sequences with dimensions of 129×512 corresponding to the NI image, and 12 coding feature sequences with dimensions of 129×512 corresponding to the TI image. Specifically, all Transformer coding blocks have the same structure, and each Transformer coding block performs the following operations on each embedded feature sequence F (which has a dimension of 129×512): (1) Perform layer normalization operation on the embedded feature sequence F to obtain the normalized feature sequence F1; (2) Perform a multi-head self-attention operation on the normalized feature sequence F1 obtained in step (1) (parameters: 12 attention heads, 64 dimensions of each attention head) to obtain a feature sequence F2 with dimension L×D; Specifically, this step involves the following steps: First, three linear transformations are used to generate a query matrix Q, a key matrix K, and a value matrix V, each with a dimension of 129×512. Then, the query matrix Q, the key matrix K, and the value matrix V are divided into 12 heads according to their feature dimensions, each head having a dimension of 129×64. Next, the attention weight Softmax((Q·K^T) / √64) for each head is obtained. Subsequently, this attention weight is multiplied by the value matrix V to obtain a weighted feature. Finally, the weighted features of the 12 heads are concatenated and then linearly projected to obtain a feature sequence F2 with a dimension of L×D. (3) Add the feature sequence F2 obtained in step (2) to the embedded feature sequence F element by element (residual connection) to obtain the feature sequence F3; (4) Perform layer normalization on the feature sequence F3 obtained in step (3) to obtain the normalized feature sequence F4; (5) Perform feedforward neural network transformation on the normalized feature sequence F4 obtained in step (4). First, expand the dimension of the normalized feature sequence F4 from D to 4D (i.e., from 768 to 3072) through a linear layer. Then, activate the normalized feature sequence after dimension expansion using the GELU activation function. Finally, compress the dimension of the normalized feature sequence after activation back from 4D to D through a linear layer to obtain the feature sequence F5. (6) Add the feature sequence F5 obtained in step (5) to the feature sequence F3 obtained in step (3) element by element (residual connection) to obtain an L×D encoded feature sequence as the output of the Transformer encoding block.

[0032] The advantage of the Transformer feature encoding layer is that it utilizes the powerful visual representation capabilities obtained from pre-training on large-scale image-text pairs by using a pre-trained CLIP model as a visual encoder. The deep Transformer encoder fully captures global and local semantic dependencies through a multi-head self-attention mechanism, providing highly discriminative feature representations for downstream re-identification tasks.

[0033] The input to the RGB modality normalization layer is the encoded feature sequence corresponding to the first visible light RGB image output by the Transformer feature encoding layer. This RGB modality normalization layer first performs batch normalization processing on the encoded feature sequence (feature dimension 512, momentum coefficient 0.1, numerical stability term ε=1×10⁻). 5 First, affine transformation is enabled, and the bias term is disabled. Then, the batch normalization calculation results are processed sequentially for within-batch mean and variance to obtain the mean μ and variance σ². Subsequently, the obtained mean μ and variance σ² are normalized using the formula (x-μ) / √(σ²+ε) (where x represents the global feature vector at position 0 in the encoded feature sequence) to obtain the normalized result F. Finally, the obtained normalized result F is subjected to affine transformation processing with a learnable scaling parameter γ (where the learnable scaling parameter γ is set to 1.0 during training initialization and is adaptively learned and updated during training through backpropagation, with a value range typically from 0.01 to 10.0, preferably with an initial value of 1.0) to obtain a normalized feature vector corresponding to the visible light RGB image with a dimension of 512.

[0034] The input to the NI modality normalization layer is the encoded feature sequence corresponding to the NI image output by the Transformer feature coding layer. This layer first performs the same processing on the encoded feature sequence as the RGB modality normalization layer mentioned above to obtain a normalized feature vector corresponding to the NI image with a dimension of 512.

[0035] The input to the TI modality normalization layer is the encoded feature sequence corresponding to the TI image output by the Transformer feature encoding layer. This layer performs the same processing on the encoded feature sequence as the RGB modality normalization layer mentioned above to obtain a normalized feature vector corresponding to the TI image with a dimension of 512.

[0036] The advantages of the above RGB modality normalization layer, NI modality normalization layer, and TI modality normalization layer are that by setting an independent batch normalization layer for each modality, the statistical distribution differences between different modalities are eliminated, making the feature vectors of each modality comparable, while retaining the discriminative information of each modality, laying the foundation for subsequent multimodal fusion and cross-modal matching.

[0037] The input to the identity classification layer consists of the normalized feature vectors corresponding to the first visible light RGB image output by the RGB modality normalization layer, the normalized feature vectors corresponding to the NI image output by the NI modality normalization layer, the normalized feature vectors corresponding to the TI image output by the TI modality normalization layer, and the normalized feature vectors obtained from the second visible light RGB image after processing by the image patch embedding layer, Transformer feature encoding layer, and RGB modality normalization layer (all feature vectors have a dimension of 512), as well as the identity label vectors corresponding to the above modal images (with a dimension equal to the batch size B). This layer performs linear classification and loss calculation on each input normalized feature vector to obtain the classification score corresponding to the first visible light RGB image. The classification score corresponding to the second visible light RGB image. Classification scores corresponding to NI images Classification score corresponding to TI image The normalized feature vector corresponding to the first visible light RGB image The normalized feature vector corresponding to the second visible light RGB image. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images and multimodal identity recognition loss ; The processing steps of the identity classification layer include the following: (1) Perform a linear transformation on the normalized feature vector corresponding to the first visible light RGB image (parameters: input dimension 512, output dimension C, where C is the total number of identity categories in the training set, with no bias term) to obtain the classification score corresponding to the first visible light RGB image. Its dimension is C.

[0038] (2) The classification score corresponding to the first visible light RGB image obtained in step (1). The identity tag corresponding to the visible light RGB image is used to obtain the RGB identity loss corresponding to the first visible light RGB image. .

[0039] Specifically, this step involves obtaining the RGB identity loss. The following formula is used:

[0040] in The model predicts that the sample belongs to the first... The probability of a class is determined by its classification score. The Middle Each component Calculated using the Softmax function, i.e. ; Let be the probability distribution of the i-th true label after label smoothing. When the category corresponds to the real identity label of the sample, ,otherwise ,in This is the label smoothing coefficient, with a value of 0.1.

[0041] (3) Normalize the feature vector corresponding to the first visible light RGB image. The RGB triplet loss corresponding to the first visible light RGB image is obtained by matching the identity tag with the first visible light RGB image. ; Specifically, this step involves obtaining the RGB triplet loss corresponding to the first visible light RGB image. The following formula is used:

[0042] in This represents the maximum Euclidean distance between the normalized feature vector of the first visible light RGB image and all positive samples in the same batch of training samples; where a positive sample is a sample with the same identity as the first visible light RGB image, and the maximum Euclidean distance is the Euclidean distance between the normalized feature vector of the sample that is furthest away from all positive samples. This represents the minimum Euclidean distance between the normalized feature vector of the first visible light RGB image and all negative samples in the same batch of training samples; where a negative sample is a sample with a different identity from the first visible light RGB image, and the minimum Euclidean distance is the Euclidean distance between the normalized feature vector of the closest negative sample and the first negative sample; the final RGB triplet loss. This represents the average loss of all triplet samples within the same batch.

[0043] Perform triplet loss calculation with the following parameters: margin boundary. For each anchor point sample within a batch, find the sample with the furthest distance among all positive samples (with the same identity) and calculate the positive distance. Find the nearest negative sample among all negative samples (different identities) and calculate the negative distance. The triplet loss is The batch average value was used to obtain the RGB triplet loss. .

[0044] (4) Loss the RGB identity corresponding to the first visible light RGB image obtained in step (2). The triplet loss corresponding to the first visible light RGB image obtained in step (3) A weighted combination is performed to obtain the modal loss corresponding to the first visible light RGB image. .

[0045] (5) Obtain the second visible light RGB image and its corresponding modal loss. And obtain the modal loss corresponding to the NI image. and the modal loss corresponding to the TI image. ; Specifically, the method for obtaining the modal loss corresponding to the second visible light RGB image in this step is exactly the same as steps (1) to (4) above, and the modal loss corresponding to the NI image is obtained. and the modal loss corresponding to the TI image. The method is exactly the same as steps (1) to (4) above, and will not be repeated here.

[0046] (6) The modal loss corresponding to the visible light RGB image obtained in step (4) Modal loss corresponding to the second visible light RGB image Modal loss corresponding to NI image and the modal loss corresponding to the TI image. Add them together to obtain the multimodal identity recognition loss. It outputs the classification score corresponding to the first visible light RGB image. The classification score corresponding to the second visible light RGB image. Classification scores corresponding to NI images Classification score corresponding to TI image The normalized feature vector corresponding to the first visible light RGB image The normalized feature vector corresponding to the second visible light RGB image. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images and multimodal identity recognition loss .

[0047] The advantages of the identity classification layer are that by combining cross-entropy loss and triplet loss, it not only optimizes the model's class discrimination ability, but also enhances the aggregation of similar samples and the separation of dissimilar samples in the feature space; label smoothing technology prevents model overfitting; and joint optimization of multimodal loss ensures that each modality feature vector has good discriminative power.

[0048] The input to the clustering memory layer is the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. and the normalized feature vectors corresponding to the identity tag vectors and NI images. and the normalized feature vectors corresponding to the identity label vectors and TI images. And identity tag vectors, as well as the RGB image memory updated in the last iteration. NI Image Memory and TI Image Memory (In this context, each image memory is initialized as a zero matrix at the start of training, with dimensions of 1.) This layer performs a clustering memory operation on all inputs to obtain the clustering constraint loss. And the updated modal image memory (i.e., the updated RGB image memory) after this iteration. NI Image Memory and TI Image Memory ) and output.

[0049] The processing of the clustering memory layer includes the following steps: (1) The normalized feature vector corresponding to the first visible light RGB image L2 normalization is performed to obtain the unit feature vector corresponding to the first visible light RGB image. ; (2) Calculate the unit eigenvector The RGB image memory updated in the previous iteration The inner product similarity of each category prototype is calculated, and the dimensions of all inner product similarities are denoted by . Similarity vector ; (3) Convert the similarity vector Divide by temperature coefficient To obtain the scaled logits vector ; (4) For the scaled logits vector Cross-entropy loss is calculated using the identity label vector corresponding to the first visible light RGB image to obtain the clustering loss corresponding to the first visible light RGB image. .

[0050] (5) Update the RGB image memory after the previous iteration. Update to obtain the RGB image memory library updated in this iteration. For each sample in the current batch, based on its identity tag Update its corresponding identity prototype: And then to The updated identity prototype in Normalization processing is performed to obtain the normalized RGB image memory updated in this iteration. .

[0051] (6) Following the same method as steps (1) to (5) above, normalize the feature vector corresponding to the NI image. Processing is performed to obtain the clustering loss corresponding to the NI image. And the NI image memory updated in this iteration And the normalized feature vector corresponding to the TI image. Processing is performed to obtain the clustering loss corresponding to the TI image. And the TI image memory updated in this iteration .

[0052] (7) The clustering loss corresponding to the first visible light RGB image obtained. NI image corresponding clustering loss The clustering constraint loss is obtained by adding the clustering loss corresponding to the TI image and the clustering loss corresponding to the TI image. ; The advantages of the clustering memory layer are that the clustering memory mechanism enables the model to learn more compact and discriminative feature representations by maintaining and dynamically updating the feature vector prototypes of each category; the temperature coefficient controls the sharpness of the similarity distribution, enhancing the model's ability to learn difficult samples; and the momentum update strategy balances historical and current information, improving the stability of prototype updates.

[0053] The input to the unbiased feature fusion layer is the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. and the normalized feature vectors corresponding to the identity tag vectors and NI images. and the normalized feature vectors corresponding to the identity label vectors and TI images. Together with the identity label vector and the unbiased fusion feature vector updated in the previous iteration. (Its initial value is a matrix of all zeros, with dimensions of ) This layer performs a clustering memory operation on all inputs to obtain an unbiased learning loss. And the unbiased memory updated in this round of iterations And output; The processing steps of the unbiased feature fusion layer include the following: (1) Normalized feature vectors corresponding to visible light RGB images Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The formula is to add elements one by one and take the average. To obtain unbiased fusion feature vectors; (2) The unbiased fusion feature vector obtained in step (1) Perform L2 normalization to obtain unit eigenvectors. ; (3) Calculate the unit eigenvector obtained in step (2). The unbiased clustering memory updated in the previous iteration The inner product similarity of each identity prototype, and the dimensions of all inner product similarities are . Similarity vector ; (4) The similarity vector obtained in step (3) Divide by temperature coefficient To obtain the scaled logits vector .

[0054] (5) The scaled logits vector obtained in step (4) Cross-entropy loss is calculated using the corresponding identity label vectors to obtain the unbiased clustering loss. .

[0055] Specifically, this step involves first processing the scaled logits vector. Apply the Softmax function to obtain the predicted probability for each identity. Then, based on the sample's real identity label Calculate the corresponding negative logarithmic probability as the loss value: .

[0056] (6) The unbiased clustering loss obtained in step (5) Multiply by unbiased weighting coefficient To obtain the weighted unbiased learning loss ; (7) Use the unit eigenvector obtained in step (2) The unbiased memory updated in the previous iteration Update to obtain the unbiased memory after this iteration. ; Specifically, this step involves first using the formula... (This update only applies to the prototype entries corresponding to the current sample.) Obtain the initially updated unbiased memory, and then perform L2 normalization on the initially updated identity prototypes to obtain the unbiased memory updated in this iteration. .

[0057] The advantages of unbiased feature fusion layers are that, through averaging the normalized feature vectors of the three modalities (RGB, NI, and TI), unbiased feature fusion learns cross-modal consistent feature representations, eliminating modality-specific biases; the introduction of an unbiased memory further constrains the discriminativeness of the fused feature vectors, enabling the model to extract common feature vectors robust to modality changes; and larger weight coefficients ( This reinforces the importance of unbiased learning and significantly improves the model's performance in cross-modal retrieval tasks.

[0058] The input to the cross-modal alignment layer is the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images (The dimension of each normalized feature vector is 1) This layer performs cross-modal contrastive learning operations on all inputs to obtain a weighted cross-modal contrastive loss. And output it.

[0059] The process of processing cross-modal alignment layers includes the following steps: (1) Based on the normalized feature vector corresponding to the first visible light RGB image Normalized feature vectors corresponding to NI images and the normalized feature vector corresponding to the TI image. Obtain the contrastive loss between pairwise normalized feature vectors; Specifically, the normalized feature vector corresponding to the first visible light RGB image is calculated. Normalized feature vectors corresponding to NI images The specific comparison loss includes: (1a) The normalized feature vector corresponding to the first visible light RGB image Normalized feature vectors corresponding to NI images L2 normalization was performed separately to obtain the unit eigenvalue matrix. and ; (1b) Calculate the unit characteristic matrix obtained in step (1a). With the unit characteristic matrix The matrix product of the transposes of the two matrices is used as the similarity matrix between them. ; (1c) The similarity matrix obtained in step (1b) Divide by temperature coefficient To obtain the logits matrix; (1d) Construct the label vector , indicating that the diagonal position is a positive sample pair, where This indicates the batch size, which is the number of samples in the current training batch.

[0060] (1e) The logits matrix obtained in step (1c) The label vector obtained from each row and step (1d) For each value in the table, cross-entropy loss is calculated to obtain the loss in the RGB to NI direction. ; (1f) The logits matrix obtained in step (1c) Each row of the transpose matrix and the tag vector obtained in step (1d) Perform cross-entropy loss calculation to obtain the loss from the NI to the RGB direction. ; (1g) Take the loss from RGB to NI direction obtained in step (1e). The loss from NI to RGB direction obtained in step (1f) The average value is used to obtain the normalized feature vector corresponding to the first visible light RGB image. Normalized feature vectors corresponding to NI images Comparison loss between ; (2) Using the method described in step (1) above (maintaining the temperature coefficient) (Unchanged), calculate the normalized feature vector corresponding to the first visible light RGB image respectively. Normalized feature vectors corresponding to TI images Cross-modal contrast loss And the normalized feature vector corresponding to the NI image. Normalized feature vectors corresponding to TI images Cross-modal contrast loss ; (3) The normalized feature vector corresponding to the first visible light RGB image obtained in step (1) Normalized feature vectors corresponding to NI images Comparison loss between and the normalized feature vector corresponding to the first visible light RGB image obtained in step (2). Normalized feature vectors corresponding to TI images Cross-modal contrast loss and the normalized feature vector corresponding to the NI image. Normalized feature vectors corresponding to TI images Cross-modal contrast loss Add to obtain the loss ; (4) The loss obtained in step (3) Multiply by cross-modal weighting coefficient To obtain weighted cross-modal contrast loss And output it.

[0061] The advantages of the cross-modal alignment layer are that cross-modal contrastive learning establishes the correspondence between modalities by bringing the feature vectors of different modalities of the same identity closer together and pushing the feature vectors of different identities further apart, which significantly improves the cross-modal retrieval performance; the temperature coefficient controls the difficulty of contrastive learning, enabling the model to learn more refined modal alignment; the comprehensive comparison of three pairs of modalities, namely RGB and NI, RGB and TI, and NI and TI, ensures good alignment between any two modalities.

[0062] The input to the hierarchical clustering constraint layer is the updated unbiased memory output by the unbiased feature fusion layer. (its dimensions are) ), human identity ID set and vehicle identity ID set And the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. Normalized feature vectors corresponding to NI images and the normalized feature vector corresponding to the TI image. and the target category label vector corresponding to the current batch of samples. This layer contains a set of human identity IDs. and vehicle identity ID set Perform the first integer neighbor clustering operation to obtain the hierarchical semantic constraint loss. And output it.

[0063] The process of hierarchical clustering constraint layer includes the following steps: (1) From unbiased memory Extract the prototype feature vector matrix for the human category. Its dimensions are ; (2) The prototype eigenvector matrix obtained in step (1) Perform FINCH hierarchical clustering (parameters: cosine distance as the distance metric, no premature termination, no output) to obtain the hierarchical clustering result matrix. (its dimensions are) , (Number of levels) and the number of clusters at each level ; (3) The hierarchical clustering result matrix obtained in step (2) Traverse all clusters at each level and obtain the mean of all prototype feature vectors in each cluster as the central feature vector of that cluster. The central feature vectors of all clusters in that level constitute the set of cluster central feature vectors for that level. (4) Calculate the average of the cluster center feature vector sets obtained in step (3) for all levels to obtain the human superclass prototype feature vector. Its dimension is 512; (5) From unbiased memory The prototype feature vector matrix of the vehicle category is obtained. Following the same parameters and methods as in steps (1) to (4), the prototype feature vector matrix of the vehicle category is subjected to FINCH hierarchical clustering to obtain the prototype feature vector of the vehicle superclass. Its dimension is 512; (6) The human superclass prototype feature vector obtained in step (4) and the vehicle superclass prototype feature vector obtained in step (5) Concatenate rows to obtain the superclass prototype matrix. Its dimensions are ; (7) Based on the target category label vector of the current batch of samples (Where a value of 0 represents a human and a value of 1 represents a vehicle), from the superclass prototype matrix obtained in step (6) Obtain the target category label vector. Corresponding batch prototype matrix Its dimensions are ; (8) Calculate the normalized feature vectors corresponding to the first visible light RGB image respectively. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The batch prototype matrix obtained in step (7) Mean square error loss between , ,as well as : in: ; ; ; in This represents the superclass prototype corresponding to each sample in the current batch (i.e., the corresponding high-level semantic prototype vector assigned according to the "human" or "vehicle" category to which the sample belongs). It serves as the target value in the loss calculation, which is used to guide the feature vectors of each modality to move closer to the center of their respective coarse-grained semantic category, thereby maintaining the semantic consistency of cross-modal features.

[0064] (9) The mean squared error loss obtained in step (8) , ,as well as Add them together to get the total loss. ; (10) The total loss obtained in step (9) Multiply by the hierarchical clustering weight coefficient To obtain the weighted hierarchical semantic constraint loss ; The advantages of the hierarchical clustering constraint layer are that the FINCH hierarchical clustering algorithm can automatically discover the semantic hierarchical structure between categories without pre-specifying the number of clusters, and has good adaptability; by aggregating categories into superclasses and constraining feature vectors to move closer to the superclass center, a higher-level semantic supervision signal is introduced, enhancing the model's ability to understand semantic structure; clustering is performed separately for human and vehicle targets, making full use of domain knowledge; mean squared error loss prompts the normalized feature vectors of each modality to follow the same semantic hierarchical structure, further enhancing cross-modal consistency.

[0065] The input to the multi-task loss optimization layer is the multimodal identity recognition loss output by the identity classification layer. Clustering constraint loss output by the clustering memory layer Unbiased learning loss output by the unbiased feature fusion layer Cross-modal contrast loss output of cross-modal alignment layer And the hierarchical clustering loss output by the hierarchical clustering constraint layer. This layer sums all the losses to obtain the total loss. Then, a mixed-precision training strategy is used to adjust the total loss. Perform backpropagation optimization to obtain the parameters of the optimized multimodal unbiased recognition model for the next batch of training: The processing steps of the multi-task loss optimization layer include the following: (1) The multimodal identity recognition loss output by the identity classification layer Clustering constraint loss output by the clustering memory layer Unbiased learning loss output by the unbiased feature fusion layer Cross-modal contrast loss output of cross-modal alignment layer And the hierarchical clustering loss output by the hierarchical clustering constraint layer. Add them together to get the total loss. And use a gradient scaler to adjust the total loss Dynamic scaling (the scaling factor is adaptively adjusted according to the training process to prevent gradient underflow in half-precision floating-point operations) is performed to obtain the scaled loss. ; (2) The scaled loss obtained in step (1) Backpropagation optimization is performed to obtain scaled gradients (i.e., numerically amplified gradient vectors) for all learnable parameters of the multimodal unbiased recognition model. (3) Determine whether the scaling gradient obtained in step (2) overflows. If it overflows, the process ends. Otherwise, restore the scaling gradient (divided by the scaling factor) to the true gradient. Based on the true gradient, use the Adam optimizer to update the parameters of the multimodal unbiased recognition model to obtain the optimized parameters of the multimodal unbiased recognition model. The specific parameters of the Adam optimizer are: base learning rate. The exponential decay rate estimated by the first moment The exponential decay rate estimated by the second moment The weight decay coefficient is 0.0005. The advantages of the multi-task loss optimization layer are that the multi-task loss function integrates five optimization objectives: identity recognition, clustering constraints, unbiased learning, cross-modal alignment, and hierarchical semantic constraints, jointly optimizing the model from different perspectives; mixed precision training significantly reduces memory usage and computation time while maintaining numerical stability; the dynamic gradient scaling strategy effectively prevents gradient underflow in half-precision operations; the warmup strategy avoids instability in the early stages of training; and the cosine annealing scheduling strategy makes the learning rate decay smoothly, which helps the model converge to a better local optimum.

[0066] The multimodal unbiased recognition model of the present invention is obtained through the following steps: (3-1) Obtain the RGBNT201 and RGBNT100 datasets as training sets, and obtain the MSVR310 dataset as a test set; Specifically, the RGBNT201 dataset is a multimodal pedestrian re-identification dataset, containing 201 pedestrian identity category image sets, each including visible light RGB images, NI images, and TI images; the RGBNT100 dataset contains 100 vehicle identity category image sets, each including visible light RGB images, NI images, and TI images; and the MSVR310 dataset contains 310 vehicle identity category image sets. The advantages of step (3-1) are as follows: First, the multimodal dataset provides three complementary imaging modalities—RGB, near-infrared, and thermal infrared—enabling the capture of target features under different lighting conditions and significantly improving the model's robustness to lighting changes. Second, the dataset covers two target classes: pedestrians and vehicles, enhancing the model's versatility and cross-class generalization ability. Finally, by fusing two datasets of different classes for joint training, the model can simultaneously learn multimodal and multi-task re-identification capabilities. (3-2) Perform data preprocessing on the training set obtained in step (3-1) to obtain the preprocessed training set; Specifically, the data preprocessing in this step includes the following operations: First, the size of all samples in the training set is uniformly adjusted to 256×128 pixels; then, each sample after size adjustment is randomly horizontally flipped with a flipping probability of 0.5; next, each sample after flipping is randomly cropped; subsequently, each sample after cropping is randomly erased with an erasure probability of 0.5; finally, a P×K sampling strategy is adopted, and the data is processed in batches. Each sample after the erasure operation is organized (where P=8 means that each batch contains 8 different identities and K=8 means that each identity contains 8 images) to obtain preprocessed samples. All preprocessed samples constitute the preprocessed training set. The advantage of step (3-2) is that data preprocessing expands the diversity of training data through enhancement methods such as image size normalization, random flipping, random cropping, and random erasing, simulating the pose changes, scale changes, and occlusion of targets in real-world scenarios; the flipping probability and erasing probability are both set to 0.5, ensuring data diversity while avoiding information distortion caused by over-enhancement; the P×K sampling strategy ensures that each batch contains multiple identities and multiple samples for each identity, providing sufficient positive and negative sample pairs for the calculation of triplet loss and clustering loss, significantly improving the model's discriminative ability; (3-3) For each sample in the preprocessed training set obtained in step (3-2) (which contains two visible light RGB images, NI image and TI image, each image has a dimension of 256×128×3), the first visible light RGB image in the sample is input into the first layer of the multimodal unbiased recognition model for image patch embedding processing, so as to output the embedded feature sequence corresponding to the first visible light RGB image of the sample, with a dimension of 129×768; (3-4) For each sample in the preprocessed training set obtained in step (3-2), the embedded feature sequence corresponding to the first visible light RGB image of the sample obtained in step (3-3) is input into the second layer of the multimodal unbiased recognition model for Transformer feature encoding to output the global feature vector corresponding to the first visible light RGB image of the sample, with a dimension of 512. (3-5) For each sample in the preprocessed training set obtained in step (3-2), the global feature vector corresponding to the first visible light RGB image of the sample obtained in step (3-4) is input into the third layer of the multimodal unbiased recognition model for batch normalization processing, so as to output the normalized feature vector corresponding to the first visible light RGB image of the sample. The dimension is 512; (3-6) For each sample in the preprocessed training set obtained in step (3-2), the NI image of the sample is successively input into the first, second, and fourth layers of the multimodal unbiased recognition model for processing, so as to output the normalized feature vector corresponding to the NI image of the sample. The dimension is 512; (3-7) For each sample in the preprocessed training set obtained in step (3-2), the TI image of the sample is sequentially input into the first, second, and fifth layers of the multimodal unbiased recognition model for processing, so as to output the normalized feature vector corresponding to the TI image of the sample. The dimension is 512; (3-8) For each sample in the preprocessed training set obtained in step (3-2), the second visible light RGB image corresponding to the sample is sequentially input into the first, second, and third layers of the multimodal unbiased recognition model for processing, so as to output the normalized feature vector corresponding to the second visible light RGB image of the sample. The dimension is 512; The advantages of steps (3-3) to (3-8) are as follows: First, by using the pre-trained CLIP model as the visual encoder, the powerful visual representation capabilities obtained from pre-training on large-scale image-text pairs are utilized, providing good feature initialization for downstream re-identification tasks; Second, the deep Transformer encoder extracts global feature vectors from the three modal images respectively, fully capturing the unique information of each modality; Finally, the batch normalization operation eliminates the statistical distribution differences between different modalities, making the feature vectors of each modality comparable. (3-9) For each sample in the preprocessed training set obtained in step (3-2), the four modality-normalized feature vectors corresponding to that sample obtained in steps (3-5), (3-6), (3-7), and (3-8) are... , , , The corresponding identity label is input into the 6th layer of the multimodal unbiased recognition model for identity classification processing to obtain the classification score corresponding to the first visible light RGB image of the sample. The classification score corresponding to the second visible light RGB image. Classification scores corresponding to NI images Classification score corresponding to TI image and multimodal identity recognition loss ; (3-10) For each sample in the preprocessed training set obtained in step (3-2), the normalized feature vector corresponding to the first visible light RGB image of that sample obtained in steps (3-5), (3-6), and (3-7) is used. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The corresponding identity label is input into the 7th layer of the multimodal unbiased recognition model for clustering memory processing to obtain the clustering constraint loss corresponding to the sample. ; (3-11) For each sample in the preprocessed training set obtained in step (3-2), the normalized feature vector corresponding to the first visible light RGB image of that sample obtained in steps (3-5), (3-6), and (3-7) is... Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The corresponding identity label is input into the 8th layer of the multimodal unbiased recognition model for unbiased feature fusion processing to obtain the unbiased learning loss corresponding to the sample. and unbiased memory bank ; (3-12) For each sample in the preprocessed training set obtained in step (3-2), the normalized feature vector corresponding to the first visible light RGB image of that sample obtained in steps (3-5), (3-6), and (3-7) is used. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The 9th layer of the input multimodal unbiased recognition model is subjected to cross-modal alignment processing to obtain the cross-modal contrast loss corresponding to the sample. ; (3-13) For each sample in the preprocessed training set obtained in step (3-2), the unbiased memory corresponding to that sample obtained in step (3-11) is... Human identity ID set and vehicle identity ID set And the three modality-normalized feature vectors corresponding to the sample obtained in steps (3-5), (3-6), and (3-7). , , The first integer neighbor clustering operation is performed on the 10th layer of the multimodal unbiased recognition model to obtain the hierarchical semantic constraint loss corresponding to the sample. ; (3-14) For each sample in the preprocessed training set obtained in step (3-2), the multimodal identity recognition loss corresponding to that sample is obtained in step (3-9). The clustering constraint loss corresponding to the sample obtained in step (3-10) The unbiased learning loss corresponding to the sample obtained in step (3-11) The cross-modal contrast loss corresponding to the sample obtained in step (3-12) The hierarchical semantic constraint loss corresponding to the sample obtained in step (3-13) Calculate the total loss corresponding to this sample. ; The advantage of step (3-14) is that the multi-task loss function integrates five optimization objectives: identity recognition, clustering constraints, unbiased learning, cross-modal alignment, and hierarchical semantic constraints. It jointly optimizes the model from different perspectives: the identity recognition loss ensures basic discriminative ability, the clustering constraint loss enhances feature compactness, the unbiased learning loss improves cross-modal consistency, the cross-modal contrast loss establishes modal correspondence, and the hierarchical semantic constraint loss introduces high-level semantic supervision. This multi-level and multi-perspective supervision strategy enables the model to learn more robust, discriminative, and generalized feature representations. (3-15) For each sample in the preprocessed training set obtained in step (3-2), based on the total loss corresponding to the sample obtained in step (3-14), a mixed precision training strategy is used to perform backpropagation and parameter update operations on the multimodal unbiased recognition model. Specifically, step (3-15) first uses a gradient scaler to... Dynamic scaling is performed, with the scaling factor adaptively adjusted based on the training process. Then, backpropagation is performed on the scaled loss to calculate the gradients of all learnable parameters of the model. Finally, the model parameters are updated using the Adam optimizer, whose parameters are: the base learning rate... The exponential decay rate estimated by the first moment The exponential decay rate estimated by the second moment The weight decay coefficient is 0.0005; then the scaling factor of the gradient scaler is updated. If gradient overflow occurs, the scaling factor is decreased; if no overflow occurs for several consecutive times, the scaling factor is increased. Finally, a warmup and cosine annealing scheduling strategy is applied to the learning rate: in the first 10 iterations of the initial training phase, the learning rate linearly increases from 0 to... During subsequent training, the learning rate follows a cosine function from... The training cycle is smoothly reduced to near 0, with a total of 50 epochs. The advantages of steps (3-15) are as follows: Mixed-precision training uses half-precision floating-point numbers (FP16) for forward propagation and gradient calculation, and single-precision floating-point numbers (FP32) to store the sovereign sum for parameter updates. This significantly reduces memory usage and computation time while maintaining numerical stability, thus improving training efficiency. The dynamic gradient scaling strategy effectively prevents gradient underflow in half-precision operations. The Adam optimizer combines the advantages of momentum and adaptive learning rates, enabling rapid convergence. The warmup strategy avoids instability caused by insufficient parameter initialization in the early stages of training through a small initial learning rate and a gradual growth mechanism, providing a smooth training start-up process for the model. The cosine annealing scheduling strategy causes the learning rate to decay smoothly according to the cosine function, which helps the model to be fine-tuned in the later stages of training and converge to a better local optimum. (3-16) For the preprocessed training set obtained in step (3-2), repeat steps (3-3) to (3-15) until the multimodal unbiased recognition model reaches the preset number of training epochs (50 epochs in this invention), and obtain the optimal parameters of the multimodal unbiased recognition model at this time, thereby obtaining the initially trained multimodal unbiased recognition model. Specifically, in step (3-16), after each epoch of training, the model is evaluated on the validation set. The evaluation metrics include the mean accuracy (mAP) and Rank-1 accuracy of six cross-modal retrieval modes (RGB query NI library R→N, NI query RGB library N→R, RGB query TI library R→T, TI query RGB library T→R, NI query TI library N→T, TI query NI library T→N) and one multimodal fusion retrieval mode (RNT→RNT). The harmonic mean of all seven mode metrics is calculated as the comprehensive performance evaluation. If the harmonic mean reaches the current optimal value, the model parameters for that epoch are saved as the best model. The advantages of step (3-16) are that a sufficient number of training epochs (50 epochs) ensures that the model fully learns the feature patterns in the training data and avoids underfitting; periodic validation evaluation can monitor model performance in a timely manner and prevent overfitting; saving the best model strategy ensures that the model parameters with the best validation performance are obtained; and using the harmonic mean, compared with the arithmetic mean, pays more attention to the balance of each mode, avoids the problem of a certain mode's low performance being masked by the average, and can better reflect the overall performance of the model. (3-17) Use the test set obtained in step (3-1) to test and verify the multimodal unbiased recognition model initially trained in step (3-16) until the recognition accuracy reaches the optimal level, thereby obtaining the final trained multimodal unbiased recognition model. Specifically, step (3-17) first loads the optimal model parameters saved in step (3-16); then sets the model to evaluation mode, disables the statistical update of the batch normalization layer and the random deactivation layer; for each query sample in the test set, it sequentially inputs layers 1 to 5 of the model to extract the normalized feature vectors of the three modalities. , , (Dimensions are all 512); For cross-modal retrieval, the cosine similarity is calculated directly using the feature vectors of the two modalities; for multimodal fusion retrieval, the feature vectors of the three modalities are concatenated. (Dimension 1536), then calculate the similarity between the fused feature vectors of the query sample and the database sample; sort all samples in the database in descending order of similarity; calculate the Rank-1 accuracy, Rank-5 accuracy, Rank-10 accuracy and mAP for seven retrieval modes (R→N, N→R, R→T, T→R, N→T, T→N, RNT→RNT); take the average of the metrics for the seven modes as the comprehensive performance evaluation; The advantages of step (3-17) are that the testing and validation phase objectively reflects the model's generalization ability and practical application potential by evaluating the model's performance on independent test sets; the comprehensive testing of seven retrieval modes verifies the model's robustness under various modal combinations; cross-modal retrieval and multimodal fusion retrieval respectively demonstrate the model's modality alignment ability and multimodal information integration ability; multiple evaluation metrics (Rank-1, Rank-5, Rank-10, mAP) measure retrieval performance from different perspectives; and the use of the harmonic mean ensures the model's balanced performance in all retrieval scenarios. Test Example 1: To verify the performance advantages of the proposed method in multimodal and multi-task re-identification tasks, comparative tests were conducted with five mainstream methods on the RGBNT201 and RGBNT100 datasets. All models were trained on a combined dataset of RGBNT201 and RGBNT100, and then evaluated on the test sets of both datasets. Test metrics included mAP and Rank-1 accuracy for six cross-modal retrieval modes (R→N, N→R, R→T, T→R, N→T, T→N) and one fusion modal retrieval mode (RNT→RNT), as well as the harmonic mean (Harm_Mean) for all seven scenarios, comprehensively evaluating model performance from multiple perspectives. Experimental results are detailed in Table 1 below.

[0067] Table 1. Performance comparison of the present invention and existing methods on the RGBNT201 dataset.

[0068] Table 1 (continued)

[0069] Table 2 Performance comparison of the present invention and existing methods on the RGBNT100 dataset.

[0070] Table 2 (continued)

[0071] The experimental results in Tables 1 and 2 demonstrate that the method of this invention has significant advantages in multimodal, multi-task re-identification tasks: On the RGBNT201 pedestrian dataset, the method of this invention significantly outperformed all comparative methods in all six cross-modal retrieval scenarios, fully validating its cross-modal matching capability. In multimodal fusion retrieval (RNT→RNT), the mAP of the method of this invention was 64.91%, only 0.8 percentage points different from the PromptMA method (65.71%), which has a specially designed multimodal fusion matching module. Taking the harmonic mean of all seven retrieval scenarios, the mAP of the method of this invention reached 20.75%, and the Rank-1 accuracy reached 19.97%, which is 5.43 and 9.02 percentage points higher than the second-best method, PromptMA, respectively, demonstrating a significant comprehensive advantage.

[0072] On the RGBNT100 vehicle dataset, the method of this invention consistently and significantly outperformed all comparable methods in all seven retrieval scenarios; its harmonic mean mAP reached 22.63% and its Rank-1 accuracy reached 27.50%, which improved by 8.70 and 16.22 percentage points respectively compared with the second best method PromptMA, further demonstrating the effectiveness of the method.

[0073] The excellent consistency across datasets demonstrates that the method of this invention has good generalization ability for different target categories (pedestrians / vehicles). This is attributed to the effective design of the Unbiased Prototype Enhancement Module (UPME) and the Class Prototype Clustering Regularization Module (CPCR), which enables the model to learn cross-modal consistency and cross-class discriminative feature representations simultaneously.

[0074] In summary, the method of this invention outperforms existing technologies in key indicators such as retrieval accuracy, cross-modal robustness, and cross-class generalization ability. It has high practical value and promotion potential, and provides an efficient and reliable technical solution for practical applications in the field of multimodal and multi-task re-identification.

[0075] Test Example 2 To fully verify the effectiveness of each component module of the method of this invention, detailed ablation experiments were conducted on the RGBNT201 and RGBNT100 datasets. An incremental ensemble approach was used, gradually adding the proposed modules to the baseline model, and the contribution of each module was verified by comparing the performance of different configurations. For simplicity, all metrics used the harmonic mean of seven retrieval scenarios. Experimental results are detailed in Table 3. Table 3 Ablation Experiment Results for Each Module

[0076] Effect analysis of the unbiased feature fusion layer (Unbiased Prototype Enhancement Module UPME): By utilizing the identity consistency information embedded in the modal unbiased prototype, the UPME module enhances the feature representation between different modalities, thereby improving cross-modal and multimodal representation capabilities. Experimental results show that after adding the UPME module, the mAP improved by 2.80 percentage points (from 13.85% to 16.65%) on the RGBNT201 dataset, and the Rank-1 accuracy improved by 3.92 percentage points; on the RGBNT100 dataset, the mAP improved by 2.01 percentage points, and the Rank-1 accuracy improved by 2.81 percentage points; the average improvement across the two datasets was 2.41 percentage points for mAP and 3.37 percentage points for Rank-1 accuracy, fully validating the effectiveness of unbiased prototype learning for cross-modal feature enhancement.

[0077] Performance analysis of the hierarchical clustering constraint layer (Category Prototype Clustering Regularization Module CPCR): The CPCR module clusters the modal unbiased prototypes generated by UPME to obtain category-consistent prototypes, and then uses their category-discriminative semantics to regularize the training process. After adding the CPCR module, on the RGBNT201 dataset, compared with the configuration using only UPME, the mAP further improved by 4.10 percentage points (from 16.65% to 20.75%), and the Rank-1 accuracy improved by 5.99 percentage points; on the RGBNT100 dataset, the mAP improved by 0.73 percentage points, and the Rank-1 accuracy improved by 2.91 percentage points; compared with the baseline model, the complete method improved the average mAP and Rank-1 accuracy by 4.82 and 7.82 percentage points, respectively, on both datasets, significantly verifying the important role of category-level semantic regularization in improving cross-category discrimination ability.

[0078] The results of this ablation experiment show that the UPME and CPCR modules each play an irreplaceable role, and their synergistic effect enables the method of this invention to achieve significant performance improvement in multimodal and multi-task re-identification tasks.

[0079] Test Example 3 To evaluate the cross-domain generalization ability of the method of this invention, cross-domain retrieval experiments were conducted. In traditional re-identification evaluation, the training and test sets are usually collected under the same scene conditions, which may lead to the learned feature space being limited to a specific environment and failing to fully reflect the generalization ability required for practical applications. Therefore, this experiment trained the model on a combined dataset of RGBNT201 and RGBNT100, and then evaluated it on the unseen MSVR310 dataset. The MSVR310 dataset contains 310 vehicle identity categories and differs significantly from the training data in terms of scene, shooting conditions, etc., which can effectively test the cross-domain generalization performance of the model. The experimental results are detailed in Table 4.

[0080] Table 4. Cross-domain generalization performance evaluation (MSVR310 dataset)

[0081] Table 5. Cross-domain generalization performance comparison (NI / TI related and fusion search)

[0082] The following important findings can be observed from the experimental results in Table 5: First, all models showed a significant decrease in detection accuracy on the MSVR310 dataset compared to the RGBNT201 and RGBNT100 datasets, indicating that cross-domain generalization remains a highly challenging problem for existing re-identification models. Factors such as scene changes, lighting conditions, and shooting angles all significantly impact model performance.

[0083] Secondly, the four methods HTT, TOP-ReID, EDITOR, and DeMo all suffered severe performance degradation in cross-modal and multimodal retrieval scenarios on the unseen MSVR310 dataset, with harmonic mean mAP all below 5% and Rank-1 accuracy all below 2%, indicating that the feature representations learned by these methods are sensitive to domain changes and have weak generalization ability.

[0084] In comparison, the proposed method and the PromptMA method achieved significantly better performance. The proposed method consistently outperformed the PromptMA method in all retrieval scenarios. The proposed method achieved a harmonic mean mAP of 11.44% and a Rank-1 accuracy of 13.77%, which are improvements of 7.16 and 10.99 percentage points respectively compared to PromptMA, and more than 8 percentage points compared to the third best method, demonstrating excellent cross-domain generalization ability.

[0085] The superior cross-domain retrieval performance of the method in this invention verifies its effectiveness from another perspective: the unbiased prototype enhancement module not only enhances the cross-modal robustness of the fused feature space but also improves its adaptability to domain changes by learning modality-consistent representations; the categorical prototype clustering regularization module strengthens the model's ability to discriminate semantic features of different categories by introducing category-level semantic supervision, making the learned feature representations more transferable. These designs enable the method of this invention to maintain stable performance in unseen new domains, providing a reliable guarantee for deployment in practical application scenarios.

[0086] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal, multi-task target re-identification method based on unbiased prototype consistency learning, characterized in that, Includes the following steps: (1) Acquire the image to be recognized; (2) Perform data preprocessing on the image to be identified obtained in step (1) to obtain the preprocessed image; (3) Input the preprocessed image obtained in step (2) into the pre-trained multimodal unbiased recognition model to obtain the feature representation of the image; (4) Compare the feature representation of the image obtained in step (3) with the features of multiple samples registered in the pre-established target library to obtain multiple similarity matching results, and sort all similarity matching results in descending order; (5) Obtain the sample feature corresponding to the highest-ranked similarity matching result in the target library from the similarity matching results obtained in step (4), and use the sample and its confidence score corresponding to the sample feature in the target library as the final recognition result.

2. The multimodal, multi-task target re-identification method based on unbiased prototype consistency learning according to claim 1, characterized in that, Step (2) is as follows: First, the size of the image to be recognized is adjusted to 256×128×3, where 256×128 represents the resolution of the image to be recognized and 3 represents the number of RGB channels; then, one or any combination of image normalization, random flipping, and random cropping is performed on the image after the size adjustment to obtain the preprocessed image. The multimodal unbiased recognition model includes an image patch embedding layer, a Transformer feature encoding layer, an RGB modality normalization layer, an NI modality normalization layer, a TI modality normalization layer, an identity classification layer, a clustering memory layer, an unbiased feature fusion layer, a cross-modal alignment layer, a hierarchical clustering constraint layer, and a multi-task loss optimization layer. The input to the image patch embedding layer consists of three modalities: a first visible light RGB image, an NI image, and a TI image. Each image has dimensions of 256×128×3. The image patch embedding layer performs the following operations on each modal image: First, it performs convolution operations on the first visible light RGB image, NI image, and TI image respectively to divide them into 128 image patches of 16×8 dimensions, thereby obtaining feature maps corresponding to the visible light RGB image, NI image, and TI image with dimensions of 16×8×768. Then, it flattens the feature maps corresponding to the first visible light RGB image, NI image, and TI image from the spatial dimension into a first visible light RGB image of length 128. The feature sequences corresponding to the first visible light RGB image, NI image, and TI image have a feature dimension of 768 at each sequence position. Next, a learnable class label with a dimension of 1×768 is inserted at the beginning of the feature sequences corresponding to the first visible light RGB image, NI image, and TI image, expanding the length of the feature sequences corresponding to the first visible light RGB image, NI image, and TI image to 129 respectively. Subsequently, the learnable positional encoding is added element-wise to the feature sequences corresponding to the first visible light RGB image, NI image, and TI image respectively. Finally, layer normalization is performed on the summed feature sequences to obtain the embedded feature sequences corresponding to the first visible light RGB image, NI image, and TI image respectively, and then output.

3. The multimodal, multi-task target re-identification method based on unbiased prototype consistency learning according to claim 2, characterized in that, The input to the Transformer feature coding layer is the embedding feature sequences corresponding to the first visible light RGB image, NI image, and TI image output by the image block embedding layer. The Transformer feature coding layer inputs the embedding feature sequences corresponding to the first visible light RGB image, NI image, and TI image to the encoder containing 12 Transformer coding blocks for deep feature extraction, so as to output 12 coding feature sequences with dimensions of 129×512 corresponding to the first visible light RGB image, 12 coding feature sequences with dimensions of 129×512 corresponding to the NI image, and 12 coding feature sequences with dimensions of 129×512 corresponding to the TI image. All Transformer coding blocks have the same structure, and each Transformer coding block performs the following operations for each embedded feature sequence F: (a1) Perform layer normalization on the embedded feature sequence F to obtain the normalized feature sequence F1; (a2) Perform a multi-head self-attention operation on the normalized feature sequence F1 obtained in step (a1) to obtain a feature sequence F2 with dimension L×D; Specifically, this step involves the following steps: First, three linear transformations are used to generate a query matrix Q, a key matrix K, and a value matrix V, each with a dimension of 129×512. Then, the query matrix Q, the key matrix K, and the value matrix V are divided into 12 heads according to their feature dimensions, each head having a dimension of 129×64. Next, the attention weight Softmax((Q·K^T) / √64) for each head is obtained. Subsequently, this attention weight is multiplied by the value matrix V to obtain a weighted feature. Finally, the weighted features of the 12 heads are concatenated and then linearly projected to obtain a feature sequence F2 with a dimension of L×D. (a3) Add the feature sequence F2 obtained in step (a2) to the embedded feature sequence F element by element to obtain the feature sequence F3; (a4) Perform layer normalization on the feature sequence F3 obtained in step (a3) ​​to obtain the normalized feature sequence F4; (a5) Perform feedforward neural network transformation on the normalized feature sequence F4 obtained in step (a4). First, expand the dimension of the normalized feature sequence F4 from 768 to 3072 through a linear layer. Then, activate the normalized feature sequence after dimension expansion using the GELU activation function. Finally, compress the dimension of the normalized feature sequence after activation back from 3072 to 768 through a linear layer to obtain the feature sequence F5. (a6) Add the feature sequence F5 obtained in step (a5) to the feature sequence F3 obtained in step (a3) ​​element by element to obtain an L×D encoded feature sequence, which is used as the output of the Transformer encoded block.

4. The multimodal, multi-task target re-identification method based on unbiased prototype consistency learning according to claim 3, characterized in that, The input to the RGB modality normalization layer is the encoded feature sequence corresponding to the first visible light RGB image output by the Transformer feature encoding layer. The RGB modality normalization layer first performs batch normalization on the encoded feature sequence. Then, the batch normalization calculation result is processed by batch mean and variance processing to obtain the mean μ and variance σ². Subsequently, the obtained mean μ and variance σ² are normalized using the formula (x-μ) / √(σ²+ε), where x represents the global feature vector at position 0 in the encoded feature sequence to obtain the normalization result F. Finally, the obtained normalization result F is processed by an affine transformation with a learnable scaling parameter γ to obtain a normalized feature vector corresponding to the visible light RGB image with a dimension of 512. The input to the NI modality normalization layer is the encoded feature sequence corresponding to the NI image output by the Transformer feature coding layer. This layer first performs the same processing on the encoded feature sequence as the RGB modality normalization layer mentioned above to obtain a normalized feature vector corresponding to the NI image with a dimension of 512. The input to the TI modality normalization layer is the encoded feature sequence corresponding to the TI image output by the Transformer feature encoding layer. This layer performs the same processing on the encoded feature sequence as the RGB modality normalization layer above to obtain a normalized feature vector corresponding to the TI image with a dimension of 512. The input to the identity classification layer consists of the normalized feature vectors corresponding to the first visible light RGB image output by the RGB modality normalization layer, the normalized feature vectors corresponding to the NI image output by the NI modality normalization layer, the normalized feature vectors corresponding to the TI image output by the TI modality normalization layer, the normalized feature vectors obtained from the second visible light RGB image after processing by the image patch embedding layer, the Transformer feature encoding layer, and the RGB modality normalization layer, and the identity label vectors corresponding to the aforementioned modalities. This layer performs linear classification and loss calculation on each input normalized feature vector to obtain the classification score corresponding to the first visible light RGB image. The classification score corresponding to the second visible light RGB image. Classification scores corresponding to NI images Classification score corresponding to TI image The normalized feature vector corresponding to the first visible light RGB image The normalized feature vector corresponding to the second visible light RGB image. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images and multimodal identity recognition loss ; The processing steps of the identity classification layer include the following: (b1) Perform a linear transformation on the normalized feature vector corresponding to the first visible light RGB image to obtain the classification score corresponding to the first visible light RGB image. Its dimension is C; (b2) The classification score corresponding to the first visible light RGB image obtained in step (b1). The identity tag corresponding to the visible light RGB image is used to obtain the RGB identity loss corresponding to the first visible light RGB image. ; Specifically, this step involves obtaining the RGB identity loss. The following formula is used: ; in The model predicts that the sample belongs to the first... The probability of a class is determined by its classification score. The Middle Each component Calculated using the Softmax function, i.e. ; Let be the probability distribution of the i-th true label after label smoothing. When the category corresponds to the real identity label of the sample, ,otherwise ,in This is the label smoothing coefficient, with a value of 0.1; (b3) The normalized feature vector corresponding to the first visible light RGB image The RGB triplet loss corresponding to the first visible light RGB image is obtained by matching the identity tag with the first visible light RGB image. ; Specifically, this step involves obtaining the RGB triplet loss corresponding to the first visible light RGB image. The following formula is used: ； in This represents the maximum Euclidean distance between the normalized feature vector of the first visible light RGB image and all positive samples in the same batch of training samples; where a positive sample is a sample with the same identity as the first visible light RGB image, and the maximum Euclidean distance is the Euclidean distance between the normalized feature vector of the sample that is furthest away from all positive samples. This represents the minimum Euclidean distance between the normalized feature vector of the first visible light RGB image and all negative samples in the same batch of training samples; where a negative sample is a sample with a different identity from the first visible light RGB image, and the minimum Euclidean distance is the Euclidean distance between the normalized feature vector of the closest negative sample and the first negative sample; the final RGB triplet loss. This is the average loss of all triplet samples within the same batch; (b4) Loss the RGB identity corresponding to the first visible light RGB image obtained in step (b2). The triplet loss corresponding to the first visible light RGB image obtained in step (b3) A weighted combination is performed to obtain the modal loss corresponding to the first visible light RGB image. ; (b5) Obtain the second visible light RGB image and its corresponding modal loss. And obtain the modal loss corresponding to the NI image. and the modal loss corresponding to the TI image. ; (b6) The modal loss corresponding to the visible light RGB image obtained in step (b4) and the modal loss corresponding to the second visible light RGB image obtained in step (b5). Modal loss corresponding to NI image and the modal loss corresponding to the TI image. Add them together to obtain the multimodal identity recognition loss. It outputs the classification score corresponding to the first visible light RGB image. The classification score corresponding to the second visible light RGB image. Classification scores corresponding to NI images Classification score corresponding to TI image The normalized feature vector corresponding to the first visible light RGB image The normalized feature vector corresponding to the second visible light RGB image. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images and multimodal identity recognition loss .

5. The multimodal, multi-task target re-identification method based on unbiased prototype consistency learning according to claim 4, characterized in that, The input to the clustering memory layer is the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. and the normalized feature vectors corresponding to the identity tag vectors and NI images. and the normalized feature vectors corresponding to the identity label vectors and TI images. And identity tag vectors, as well as the RGB image memory updated in the last iteration. NI Image Memory and TI Image Memory This layer performs a clustering memory operation on all inputs to obtain the clustering constraint loss. and the updated RGB image memory library NI Image Memory and TI Image Memory And output; The processing of the clustering memory layer includes the following steps: (c1) The normalized feature vector corresponding to the first visible light RGB image L2 normalization is performed to obtain the unit feature vector corresponding to the first visible light RGB image. ; (c2) Calculate the unit eigenvector The RGB image memory updated in the previous iteration The inner product similarity of each category prototype is calculated, and the dimensions of all inner product similarities are denoted by . Similarity vector ; (c3) Transfer the similarity vector Divide by temperature coefficient To obtain the scaled logits vector ; (c4) For the scaled logits vector Cross-entropy loss is calculated using the identity label vector corresponding to the first visible light RGB image to obtain the clustering loss corresponding to the first visible light RGB image. ; (c5) Update the RGB image memory after the previous iteration Update to obtain the RGB image memory library updated in this iteration. For each sample in the current batch, based on its identity tag Update its corresponding identity prototype: And then to The updated identity prototype in Normalization processing is performed to obtain the normalized RGB image memory updated in this iteration. ; (c6) Following the same method as steps (c1) to (c5) above, normalize the feature vector corresponding to the NI image. Processing is performed to obtain the clustering loss corresponding to the NI image. And the NI image memory updated in this iteration And the normalized feature vector corresponding to the TI image. Processing is performed to obtain the clustering loss corresponding to the TI image. And the TI image memory updated in this round of iterations ; (c7) The clustering loss corresponding to the first visible light RGB image obtained NI image corresponding clustering loss The clustering constraint loss is obtained by adding the clustering loss corresponding to the TI image and the clustering loss corresponding to the TI image. .

6. The multimodal, multi-task target re-identification method based on unbiased prototype consistency learning according to claim 5, characterized in that, The input to the unbiased feature fusion layer is the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. and the normalized feature vectors corresponding to the identity tag vectors and NI images. and the normalized feature vectors corresponding to the identity label vectors and TI images. Together with the identity label vector and the unbiased fusion feature vector updated in the previous iteration. Its initial value is a matrix of all zeros with dimension . This layer performs a clustering memory operation on all inputs to obtain an unbiased learning loss. And the unbiased memory updated in this round of iterations And output; The processing steps of the unbiased feature fusion layer include the following: (d1) Normalized feature vector corresponding to the visible light RGB image Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The formula is to add elements one by one and take the average. To obtain unbiased fused feature vectors; (d2) The unbiased fusion feature vector obtained in step (d1) Perform L2 normalization to obtain unit eigenvectors. ; (d3) Calculate the unit eigenvector obtained in step (d2). The unbiased clustering memory updated in the previous iteration The inner product similarity of each identity prototype, and the dimensions of all inner product similarities are: Similarity vector ; (d4) The similarity vector obtained in step (d3) Divide by temperature coefficient To obtain the scaled logits vector ; (d5) The scaled logits vector obtained in step (d4) Cross-entropy loss is calculated using the corresponding identity label vectors to obtain the unbiased clustering loss. ; Specifically, this step involves first processing the scaled logits vector. Apply the Softmax function to obtain the predicted probability for each identity. Then, based on the sample's real identity label Calculate the corresponding negative logarithmic probability as the loss value: ; (d6) Apply the unbiased clustering loss obtained in step (d5) Multiply by unbiased weighting coefficient To obtain the weighted unbiased learning loss ; (d7) Use the unit eigenvector obtained in step (d2) The unbiased memory updated in the previous iteration Update to obtain the unbiased memory after this iteration. .

7. The multimodal, multi-task target re-identification method based on unbiased prototype consistency learning according to claim 6, characterized in that, The input to the cross-modal alignment layer is the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images This layer performs cross-modal contrastive learning on all inputs to obtain a weighted cross-modal contrastive loss. And output; The process of processing cross-modal alignment layers includes the following steps: (e1) Based on the normalized feature vector corresponding to the first visible light RGB image Normalized feature vectors corresponding to NI images and the normalized feature vector corresponding to the TI image. Obtain the contrastive loss between pairwise normalized feature vectors; Specifically, the normalized feature vector corresponding to the first visible light RGB image is calculated. Normalized feature vectors corresponding to NI images The specific comparison loss includes: (e1a) The normalized feature vector corresponding to the first visible light RGB image Normalized feature vectors corresponding to NI images L2 normalization was performed separately to obtain the unit eigenvalue matrix. and ; (e1b) The unit characteristic matrix obtained in calculation step (e1a) With the unit characteristic matrix The matrix product of the transposes of the two matrices is used as the similarity matrix between them. ; (e1c) The similarity matrix obtained in step (e1b) Divide by temperature coefficient To obtain the logits matrix; (e1d) Construct the label vector , indicating that the diagonal position is a positive sample pair, where This indicates the batch size, which is the number of samples in the current training batch; (e1e) is the logits matrix obtained in step (e1c). The label vector obtained from each row and step (e1d) For each value in the table, cross-entropy loss is calculated to obtain the loss in the RGB to NI direction. ; (e1f) The logits matrix obtained in step (e1c) Each row of the transpose matrix and the tag vector obtained in step (e1d) Perform cross-entropy loss calculation to obtain the loss from the NI to the RGB direction. ; (e1g) Take the loss from RGB to NI direction obtained in step (e1e). The loss from NI to RGB direction obtained in step (e1f) The average value is used to obtain the normalized feature vector corresponding to the first visible light RGB image. Normalized feature vectors corresponding to NI images Comparison loss between ; (e2) Using the method described in step (e1) above, calculate the normalized feature vector corresponding to the first visible light RGB image. Normalized feature vectors corresponding to TI images Cross-modal contrast loss And the normalized feature vector corresponding to the NI image. Normalized feature vectors corresponding to TI images Cross-modal contrast loss ; (e3) The normalized feature vector corresponding to the first visible light RGB image obtained in step (e1) Normalized feature vectors corresponding to NI images Comparison loss between and the normalized feature vector corresponding to the first visible light RGB image obtained in step (e2). Normalized feature vectors corresponding to TI images Cross-modal contrast loss and the normalized feature vector corresponding to the NI image. Normalized feature vectors corresponding to TI images Cross-modal contrast loss Add to obtain the loss ; (e4) The loss obtained in step (e3) Multiply by cross-modal weighting coefficient To obtain weighted cross-modal contrast loss And output it.

8. The multimodal, multi-task target re-identification method based on unbiased prototype consistency learning according to claim 7, characterized in that, The input to the hierarchical clustering constraint layer is the updated unbiased memory output by the unbiased feature fusion layer. Human identity ID set and vehicle identity ID set And the normalized feature vector corresponding to the first visible light RGB image output by the identity classification layer. Normalized feature vectors corresponding to NI images and the normalized feature vector corresponding to the TI image. and the target category label vector corresponding to the current batch of samples. This layer contains a set of human identity IDs. and vehicle identity ID set Perform the first integer neighbor clustering operation to obtain the hierarchical semantic constraint loss. And output; The process of hierarchical clustering constraint layer includes the following steps: (f1) from unbiased memory Extract the prototype feature vector matrix for the human category. Its dimensions are ; (f2) The prototype eigenvector matrix obtained in step (f1) Perform FINCH hierarchical clustering (parameters: cosine distance as the distance metric, no premature termination, no output) to obtain the hierarchical clustering result matrix. and the number of clusters at each level ; (f3) The hierarchical clustering result matrix obtained in step (f2) Traverse all clusters at each level and obtain the mean of all prototype feature vectors in each cluster as the central feature vector of that cluster. The central feature vectors of all clusters in that level constitute the set of cluster central feature vectors for that level. (f4) Calculate the average of the cluster center feature vector sets obtained in step (f3) for all levels to obtain the human superclass prototype feature vector. Its dimension is 512; (f5) From unbiased memory The prototype feature vector matrix of the vehicle category is obtained. Following the same parameters and methods as steps (f1) to (f4), FINCH hierarchical clustering is performed on the prototype feature vector matrix of the vehicle category to obtain the prototype feature vector of the vehicle superclass. Its dimension is 512; (f6) The human superclass prototype feature vector obtained in step (f4) The vehicle superclass prototype feature vector obtained in step (f5) Concatenate rows to obtain the superclass prototype matrix. Its dimensions are ; (f7) Based on the target category label vector of the current batch of samples The superclass prototype matrix obtained from step (f6) Obtain the target category label vector. Corresponding batch prototype matrix Its dimensions are ; (f8) Calculate the normalized feature vectors corresponding to the first visible light RGB image respectively. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The batch prototype matrix obtained in step (f7) Mean square error loss between , ,as well as : in: ； ； ； in This represents the superclass prototype corresponding to each sample in the current batch; (f9) The mean squared error loss obtained in step (f8) , ,as well as Add them together to get the total loss. ; (f10) The total loss obtained in step (f9) Multiply by the hierarchical clustering weight coefficient To obtain the weighted hierarchical semantic constraint loss ; The input to the multi-task loss optimization layer is the multimodal identity recognition loss output by the identity classification layer. Clustering constraint loss output by the clustering memory layer Unbiased learning loss output by the unbiased feature fusion layer Cross-modal contrast loss output of cross-modal alignment layer And the hierarchical clustering loss output by the hierarchical clustering constraint layer. This layer sums all the losses to obtain the total loss. Then, a mixed-precision training strategy is used to adjust the total loss. Perform backpropagation optimization to obtain the parameters of the optimized multimodal unbiased recognition model for the next batch of training: The processing steps of the multi-task loss optimization layer include the following: (g1) The multimodal identity recognition loss output by the identity classification layer Clustering constraint loss output by the clustering memory layer Unbiased learning loss output by the unbiased feature fusion layer Cross-modal contrast loss output of cross-modal alignment layer And the hierarchical clustering loss output by the hierarchical clustering constraint layer. Add them together to get the total loss. And use a gradient scaler to adjust the total loss Perform dynamic scaling to capture the loss after scaling. ; (g2) The scaled loss obtained in step (g1) Perform backpropagation optimization to obtain the scaling gradients of all learnable parameters of the multimodal unbiased recognition model; (g3) Determine whether the scaling gradient obtained in step (g2) overflows. If overflow occurs, the process ends. Otherwise, restore the scaling gradient to the true gradient. Based on the true gradient, use the Adam optimizer to update the parameters of the multimodal unbiased recognition model to obtain the optimized parameters of the multimodal unbiased recognition model. The specific parameters of the Adam optimizer are: base learning rate. The exponential decay rate estimated by the first moment The exponential decay rate estimated by the second moment The weight decay coefficient is 0.0005.

9. The multimodal, multi-task target re-identification method based on unbiased prototype consistency learning according to claim 8, characterized in that, The multimodal unbiased recognition model is trained through the following steps: (3-1) Obtain the RGBNT201 and RGBNT100 datasets as training sets, and obtain the MSVR310 dataset as a test set; (3-2) Perform data preprocessing on the training set obtained in step (3-1) to obtain the preprocessed training set; (3-3) For each sample in the preprocessed training set obtained in step (3-2), the first visible light RGB image in the sample is input into the first layer of the multimodal unbiased recognition model for image patch embedding processing, so as to output the embedded feature sequence corresponding to the first visible light RGB image of the sample, with a dimension of 129×768. Each sample contains two visible light RGB images, NI image and TI image, and each image has a dimension of 256×128×3. (3-4) For each sample in the preprocessed training set obtained in step (3-2), the embedded feature sequence corresponding to the first visible light RGB image of the sample obtained in step (3-3) is input into the second layer of the multimodal unbiased recognition model for Transformer feature encoding to output the global feature vector corresponding to the first visible light RGB image of the sample. (3-5) For each sample in the preprocessed training set obtained in step (3-2), the global feature vector corresponding to the first visible light RGB image of the sample obtained in step (3-4) is input into the third layer of the multimodal unbiased recognition model for batch normalization processing, so as to output the normalized feature vector corresponding to the first visible light RGB image of the sample. The dimension is 512; (3-6) For each sample in the preprocessed training set obtained in step (3-2), the NI image of the sample is successively input into the first, second, and fourth layers of the multimodal unbiased recognition model for processing, so as to output the normalized feature vector corresponding to the NI image of the sample. The dimension is 512; (3-7) For each sample in the preprocessed training set obtained in step (3-2), the TI image of the sample is sequentially input into the first, second, and fifth layers of the multimodal unbiased recognition model for processing, so as to output the normalized feature vector corresponding to the TI image of the sample. The dimension is 512; (3-8) For each sample in the preprocessed training set obtained in step (3-2), the second visible light RGB image corresponding to the sample is sequentially input into the first, second, and third layers of the multimodal unbiased recognition model for processing, so as to output the normalized feature vector corresponding to the second visible light RGB image of the sample. The dimension is 512; (3-9) For each sample in the preprocessed training set obtained in step (3-2), the four modality-normalized feature vectors corresponding to that sample obtained in steps (3-5), (3-6), (3-7), and (3-8) are... , , , The corresponding identity label is input into the 6th layer of the multimodal unbiased recognition model for identity classification processing to obtain the classification score corresponding to the first visible light RGB image of the sample. The classification score corresponding to the second visible light RGB image. Classification scores corresponding to NI images Classification score corresponding to TI image and multimodal identity recognition loss ; (3-10) For each sample in the preprocessed training set obtained in step (3-2), the normalized feature vector corresponding to the first visible light RGB image of that sample obtained in steps (3-5), (3-6), and (3-7) is used. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The corresponding identity label is input into the 7th layer of the multimodal unbiased recognition model for clustering memory processing to obtain the clustering constraint loss corresponding to the sample. ; (3-11) For each sample in the preprocessed training set obtained in step (3-2), the normalized feature vector corresponding to the first visible light RGB image of that sample obtained in steps (3-5), (3-6), and (3-7) is... Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The corresponding identity label is input into the 8th layer of the multimodal unbiased recognition model for unbiased feature fusion processing to obtain the unbiased learning loss corresponding to the sample. and unbiased memory bank ; (3-12) For each sample in the preprocessed training set obtained in step (3-2), the normalized feature vector corresponding to the first visible light RGB image of that sample obtained in steps (3-5), (3-6), and (3-7) is used. Normalized feature vectors corresponding to NI images Normalized feature vectors corresponding to TI images The 9th layer of the input multimodal unbiased recognition model is subjected to cross-modal alignment processing to obtain the cross-modal contrast loss corresponding to the sample. ; (3-13) For each sample in the preprocessed training set obtained in step (3-2), the unbiased memory corresponding to that sample obtained in step (3-11) is... Human identity ID set and vehicle identity ID set And the three modality-normalized feature vectors corresponding to the sample obtained in steps (3-5), (3-6), and (3-7). , , The first integer neighbor clustering operation is performed on the 10th layer of the multimodal unbiased recognition model to obtain the hierarchical semantic constraint loss corresponding to the sample. ; (3-14) For each sample in the preprocessed training set obtained in step (3-2), the multimodal identity recognition loss corresponding to that sample is obtained in step (3-9). The clustering constraint loss corresponding to the sample obtained in step (3-10) The unbiased learning loss corresponding to the sample obtained in step (3-11) The cross-modal contrast loss corresponding to the sample obtained in step (3-12) The hierarchical semantic constraint loss corresponding to the sample obtained in step (3-13) Calculate the total loss corresponding to this sample. ; (3-15) For each sample in the preprocessed training set obtained in step (3-2), based on the total loss corresponding to the sample obtained in step (3-14), a mixed precision training strategy is used to perform backpropagation and parameter update operations on the multimodal unbiased recognition model. (3-16) For the preprocessed training set obtained in step (3-2), repeat steps (3-3) to (3-15) until the multimodal unbiased recognition model reaches the preset number of training cycles, and obtain the optimal parameters of the multimodal unbiased recognition model at this time, so as to obtain the initially trained multimodal unbiased recognition model. (3-17) Use the test set obtained in step (3-1) to test and verify the multimodal unbiased recognition model initially trained in step (3-16) until the recognition accuracy reaches the optimal level, thereby obtaining the final trained multimodal unbiased recognition model.

10. A multimodal, multi-task target re-identification system based on unbiased prototype consistency learning, characterized in that, Includes the following modules: The first module is used to acquire the image to be recognized; The second module is used to preprocess the image to be identified obtained by the first module to obtain a preprocessed image. The third module is used to input the preprocessed image obtained from the second module into the pre-trained multimodal unbiased recognition model to obtain the feature representation of the image. The fourth module is used to compare the feature representation of the image obtained by the third module with the features of multiple samples registered in the pre-established target library to obtain multiple similarity matching results, and sort all similarity matching results in descending order; The fifth module is used to obtain the sample features corresponding to the highest-ranked similarity matching result obtained from the fourth module in the target database, and use the sample features and their confidence scores in the target database as the final recognition result.

Images