Multi-modal target re-identification method and multi-modal target re-identification model training method

Abstract

The application discloses a kind of multimodal target re-identification method, multimodal target re-identification model training method, belong to multimodal data processing technical field, the method includes: obtaining the local feature and global feature of two query modalities of the i th sample;Calculate the cross attention of Prompt feature vector and the global feature of first query modality, obtain fusion feature;Calculate the cross attention and self-attention between fusion feature and the local feature of first query modality, obtain the first fine-grained feature of first query modality;Based on the similarity score of first fine-grained feature and the local feature of second query modality and SoftMatch weight, obtain the cross-modal completion feature of first query modality.The method can realize accurate and effective multimodal target re-identification.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing technology, and in particular to a multimodal target re-identification method and a multimodal target re-identification model training method. Background Technology

[0002] Target re-identification aims to identify and match images of the same target individual across multiple camera viewpoints, and is a crucial technological component in fields such as intelligent surveillance, public safety, and urban management. Traditional target re-identification methods primarily rely on visible light images, using deep neural networks to model and compare image features. However, in practical applications, due to factors such as limited viewpoints, lighting variations, and occlusion, visible light image information of the target is often difficult to obtain completely or is of poor quality, leading to a decrease in recognition accuracy. In such cases, various auxiliary information such as sketches and text descriptions are typically used as query methods for cross-modal target re-identification, becoming an important supplementary means when visual queries are unavailable or difficult to obtain.

[0003] In recent years, various methods for cross-modal target re-identification have been proposed in related research. For example, some methods are based on modal alignment, which improves cross-modal retrieval performance through multimodal feature alignment and fusion; others are based on image-text joint learning, which enhances the relevance between text and images by utilizing multimodal attention mechanisms; in addition, for target retrieval from sketches to images, some studies have proposed end-to-end feature mapping methods, which effectively improve the recognition accuracy of sketch queries.

[0004] While existing technologies can mitigate the representational differences between modalities to some extent, they still have several limitations. First, most current methods design and train models independently for specific modalities or tasks, such as developing different network structures for text retrieval and sketch retrieval. This approach performs well in single-task scenarios, but in real-world applications, user queries may be arbitrary modalities or combinations of multiple modalities, making it difficult for existing methods to adapt to the complex needs of modal uncertainty and multi-task coexistence. Second, different modalities exhibit asymmetry in information representation and discriminative capabilities. Text can describe high-level semantics such as color and accessories, but its expression of structural details is limited; sketches can provide structural outlines and spatial relationships, but struggle to reflect semantic attributes. In actual retrieval processes, single-modal methods cannot simultaneously leverage the advantages of both, resulting in limited recognition performance.

[0005] Furthermore, current multimodal target re-identification methods typically employ static fusion or averaging strategies, failing to dynamically allocate information weights for each modality based on specific instances. This hinders the full utilization of the complementarity and discriminative power of multimodal data. Moreover, due to the high uncertainty and diversity of query modalities in real-world scenarios, existing solutions struggle to simultaneously accommodate multiple input types, such as text, sketches, and their combinations, impacting both the model's generalization ability and practical deployment efficiency. Therefore, achieving unified support for multiple modal inputs, fully mining the complementary information between different modalities, and realizing dynamic adaptive information fusion are crucial technical challenges that urgently need to be addressed in the field of multimodal target re-identification.

[0006] In summary, existing target re-identification technologies still suffer from prominent problems such as poor model compatibility, limited generalization ability, and insufficient information fusion when facing practical needs such as multimodal queries, modal uncertainty, and multi-task fusion. Therefore, there is an urgent need to propose a novel target re-identification method that can uniformly support multiple modal inputs, dynamically fuse multimodal information, and take into account the needs of multiple tasks, so as to improve the recognition accuracy and practicality of the system in complex application scenarios. Summary of the Invention

[0007] This invention provides a multimodal target re-identification method and a multimodal target re-identification model training method, which can achieve accurate and effective multimodal target re-identification. The technical solution includes at least the following: Firstly, a multimodal target re-identification method is provided, comprising: acquiring local and global features of two query modalities for an i-th sample, wherein the i-th sample includes a first query modality and a second query modality, wherein the first query modality and the second query modality are respectively one of a text modality and a sketch image modality, and the first query modality and the second query modality are different; calculating the cross-attention between the Prompt feature vector and the global features of the first query modality to obtain a fusion feature, wherein the Prompt feature vector is a learnable feature vector; calculating the cross-attention and self-attention between the fusion feature and the local features of the first query modality to obtain a first fine-grained feature of the first query modality; acquiring cross-modal completion features of the first query modality based on the similarity score and SoftMatch weight of the first fine-grained feature and the local features of the second query modality; and performing a residual connection between the cross-modal completion features of the first query modality and the global features of the first query modality to obtain a single-modal retrieval feature of the first query modality, wherein the single-modal retrieval feature of the first query modality is used for target re-identification.

[0008] Optionally, obtaining the cross-modal completion features of the first query modality based on the similarity scores and SoftMatch weights of the first fine-grained features and the local features of the second query modality includes: calculating the similarity scores and SoftMatch weights between each position of the first fine-grained features and the local features of the second query modality; selecting the features of the Top R positions based on the similarity scores; and performing weighted aggregation on the features of the Top R positions based on the SoftMatch weights of the features of the Top R positions to obtain the cross-modal completion features of the first query modality.

[0009] Optionally, the method further includes: obtaining the single-modal retrieval features of the second query modality in the same manner as the first query modality; obtaining the dynamic fusion weights of the first query modality and the second query modality based on the single-modal retrieval features of the first query modality and the second query modality; and performing weighted aggregation on the single-modal retrieval features of the first query modality and the second query modality based on the dynamic fusion weights to obtain multimodal joint features, wherein the multimodal joint features are used for multimodal target re-identification.

[0010] Secondly, a method for training a multimodal target re-identification model is provided. The i-th sample includes a target query modality, which is a visible light image modality. The method includes: acquiring global features of the target query modality; constructing a loss function for the multimodal target re-identification model; and training the multimodal target re-identification model using the i-th sample based on the loss function. The loss function includes: a first contrast loss, a second contrast loss, a third contrast loss, a consistency loss, and an alignment loss. The first contrast loss is obtained based on a first InfoNCE contrast loss between the unimodal retrieval features of a first query modality and the global features of the target query modality. The second contrast loss is obtained based on the unimodal retrieval features of a second query modality and the global features of the target query modality. The second InfoNCE comparison loss is obtained between the global features of the query modality, the third comparison loss is the third InfoNCE comparison loss between the multimodal joint features and the global features of the target query modality, the consistency loss is the normalized similarity distribution of the single-modal retrieval features of the first query modality and the single-modal retrieval features of the second query modality with respect to the global features of the target query modality, and the alignment loss is used to promote the alignment of the cross-modal completion features of the first query modality with the first fine-grained features; wherein, the single-modal retrieval features of the first query modality, the single-modal retrieval features of the second query modality, the multimodal joint features, the cross-modal completion features of the first query modality, and the first fine-grained features are obtained using the method described in the first aspect.

[0011] Optionally, the first contrast loss and the second contrast loss are obtained as follows: based on the first InfoNCE contrast loss, a confidence score for the first query modality is generated; based on the second InfoNCE contrast loss, a confidence score for the second query modality is generated; based on the confidence scores of the first and second query modalities, a weighting coefficient for the first query modality and a weighting coefficient for the second query modality are calculated; based on the weighting coefficient of the first query modality, the first InfoNCE contrast loss is weighted to obtain the first contrast loss; based on the weighting coefficient of the second query modality, the second InfoNCE contrast loss is weighted to obtain the second contrast loss.

[0012] Thirdly, a multimodal target re-identification device is also provided, comprising: an acquisition module, configured to acquire local and global features of two query modalities of an i-th sample, wherein the i-th sample includes a first query modality and a second query modality, wherein the first query modality and the second query modality are respectively one of a text modality and a sketch image modality, and the first query modality and the second query modality are different; a first attention module, configured to calculate the cross-attention of the Prompt feature vector with the global features of the first query modality to obtain fused features, wherein the Prompt feature vector is a learnable feature vector; and a second attention module, configured to calculate... The first fine-grained feature of the first query modality is obtained by calculating the cross-attention and self-attention between the fused feature and the local feature of the second query modality; the cross-modal completion module is used to obtain the cross-modal completion feature of the first query modality based on the similarity score and SoftMatch weight of the first fine-grained feature and the local feature of the second query modality; the residual connection module is used to perform residual connection between the cross-modal completion feature of the first query modality and the global feature of the first query modality to obtain the single-modal retrieval feature of the first query modality, which is used for target re-identification.

[0013] Optionally, the cross-modal completion module is further configured to calculate the similarity score and SoftMatch weight between each position of the first fine-grained feature and the local feature of the second query modality; based on the similarity score, the top R positions of features are selected, and the top R positions of features are weighted and aggregated based on the SoftMatch weight of the features of the top R positions to obtain the cross-modal completion feature of the first query modality.

[0014] Optionally, the device further includes: a multimodal joint module, configured to obtain the single-modal retrieval features of the second query modality in the same manner as the first query modality; obtain dynamic fusion weights of the first query modality and the second query modality based on the single-modal retrieval features of the first query modality and the single-modal retrieval features of the second query modality; and perform weighted aggregation on the single-modal retrieval features of the first query modality and the single-modal retrieval features of the second query modality based on the dynamic fusion weights to obtain multimodal joint features, wherein the multimodal joint features are used for multimodal target re-identification.

[0015] Fourthly, a multimodal target re-identification model training device is also provided. The i-th sample includes a target query modality, wherein the target query modality is a visible light image modality. The device includes: an acquisition module for acquiring global features of the target query modality; a loss function construction module for constructing a loss function for the multimodal target re-identification model; and a training module for training the multimodal target re-identification model using the i-th sample and based on the loss function. The loss function includes: a first contrast loss, a second contrast loss, a third contrast loss, a consistency loss, and an alignment loss. The first contrast loss is obtained based on a first InfoNCE contrast loss between the single-modal retrieval features of the first query modality and the global features of the target query modality. The second contrast loss is based on the single-modal retrieval features of the second query modality. The second InfoNCE comparison loss is obtained between the retrieval features and the global features of the target query modality. The third comparison loss is obtained between the multimodal joint features and the global features of the target query modality. The consistency loss is obtained by normalizing the similarity distribution of the single-modal retrieval features of the first query modality and the single-modal retrieval features of the second query modality to the global features of the target query modality. The alignment loss is used to promote the alignment of the cross-modal completion features of the first query modality with the first fine-grained features. The single-modal retrieval features of the first query modality, the single-modal retrieval features of the second query modality, the multimodal joint features, the cross-modal completion features of the first query modality, and the first fine-grained features are obtained by the method described in the first aspect.

[0016] In the loss function construction module, the first contrastive loss and the second contrastive loss are obtained as follows: based on the first InfoNCE contrastive loss, a confidence score for the first query mode is generated; based on the second InfoNCE contrastive loss, a confidence score for the second query mode is generated; based on the confidence scores of the first and second query modes, weighting coefficients for the first and second query modes are calculated; the first InfoNCE contrastive loss is weighted based on the weighting coefficients of the first query mode to obtain the first contrastive loss; the second InfoNCE contrastive loss is weighted based on the weighting coefficients of the second query mode to obtain the second contrastive loss.

[0017] Fifthly, a computer device is also provided, comprising: a memory and a processor, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to perform the multimodal target re-identification method and the multimodal target re-identification model training method described in the above embodiments.

[0018] In a sixth aspect, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to perform the multimodal target re-identification method and the multimodal target re-identification model training method described in the above embodiments.

[0019] In a seventh aspect, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect.

[0020] The beneficial effects of the technical solution provided by this invention include at least the following: In this embodiment, local and global features of two query modalities of the i-th sample are obtained. The i-th sample includes a first query modality and a second query modality, which are one of the text modality and the sketch image modality, respectively, and are different from each other. The cross-attention between the Prompt feature vector and the global features of the first query modality is calculated to obtain the fused features. The Prompt feature vector is a learnable feature vector. The cross-attention and self-attention between the fused features and the local features of the first query modality are calculated to obtain the first fine-grained features of the first query modality. Based on the similarity score between the first fine-grained features and the local features of the second query modality and the SoftMatch weight, the cross-modal completion features of the first query modality are obtained. The cross-modal completion features of the first query modality are residually connected with the global features of the first query modality to obtain the single-modal retrieval features of the first query modality. The single-modal retrieval features of the first query modality are used for target re-identification. The single-modal retrieval features of the first query modality not only contain the information of the first query modality itself, but also explicitly integrate the supplementary information of the second query modality, realizing information complementarity and discrimination enhancement in multimodal scenarios, thereby enabling accurate and effective multimodal target re-identification. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this embodiment, the accompanying drawings used in the description of the embodiment will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a multimodal target re-identification method provided by an exemplary embodiment of the present invention is shown; Figure 2 This is a schematic diagram of the entire process of the multimodal target re-identification algorithm; Figure 3 A flowchart of a multimodal target re-identification model training method provided by an exemplary embodiment of the present invention is shown; Figure 4 This is a schematic diagram of the structure of a multimodal target re-identification model; Figure 5 A schematic diagram of the structure of a multimodal target re-identification device provided in an exemplary embodiment of the present invention is shown; Figure 6 This diagram illustrates the structure of a multimodal target re-identification model training device provided in an exemplary embodiment of the present invention. Figure 7 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0023] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the elements or objects preceding “comprising” or “including” encompass the elements or objects listed following “comprising” or “including” and their equivalents, but do not exclude other elements or objects.

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0025] Figure 1 A flowchart of a multimodal target re-identification method provided by an exemplary embodiment of the present invention is shown, which can be executed by a computer device. Figure 2 This is a schematic diagram of the entire process of the multimodal target re-identification algorithm.

[0026] See Figure 1-2 The method includes: In step 101, the local and global features of the two query modalities of the i-th sample are obtained.

[0027] The i-th sample includes a first query modality and a second query modality. The first query modality and the second query modality are one of the text modality and the sketch image modality, respectively. The first query modality and the second query modality are different.

[0028] For example, the first query modality is a text modality and the second query modality is a sketch image modality; or, the first query modality is a sketch image modality and the second query modality is a text modality.

[0029] In this embodiment, the i-th sample can be a sample used to train the multimodal target re-identification model, or the i-th sample can be a sample used to input into the multimodal target re-identification model for querying.

[0030] When the i-th sample is used to train the multimodal object re-identification model, the i-th sample also includes the target query modality, which is a visible light image modality (e.g., an RGB image). The target query modality of the i-th sample corresponds to the first query modality and the second query modality of the i-th sample; in other words, the correct query results of the first query modality and the second query modality of the i-th sample are the target query modality of the i-th sample.

[0031] In step 101, the first query modality and the second query modality can be input into the pre-trained CLIP (Contrastive Language-Image Pre-training) model to obtain the local features of the first query modality. and global features Local features of the second query modality and global features There are many methods for implementing pre-trained CLIP models, which will not be detailed here.

[0032] In step 102, the cross-attention between the Prompt feature vector and the global features of the first query modality is calculated to obtain the fused features.

[0033] The Prompt feature vector is a learnable feature vector.

[0034] In this embodiment, during the training of the multimodal target re-identification model, the model simultaneously learns the Prompt feature vector. After the multimodal target re-identification model is trained, the Prompt feature vector is also learned, and at this point, the learned Prompt feature vector can be directly used to execute step 102.

[0035] For example, step 102 can be represented by formula (1).

[0036] (1) In formula (1), As a feature of fusion, This indicates the calculation of cross-attention between a and b. The Prompt feature vector This represents the global features of the first query modality.

[0037] When calculating the cross-attention between the Prompt feature vector and the global features of the first query modality, the Prompt feature vector is used as the query, and the global features of the first query modality are used as the key and value.

[0038] This fused feature is equivalent to the prompt of the target of interest after global feature adjustment by the first query modality, and can be used as a query in step 103.

[0039] In step 103, the cross-attention and self-attention between the fused features and the local features of the first query modality are calculated to obtain the first fine-grained features of the first query modality.

[0040] When calculating the cross-attention between the fused features and the local features of the first query modality, the fused features are used as the query, and the local features of the first query modality are used as the pieces and values.

[0041] For example, step 103 can be represented by formula (2).

[0042] (2) In formula (2), This is the first fine-grained feature. For local features of the first query modality, This indicates the calculation of self-attention. The meanings of the other parameters in formula (2) are the same as those in formula (1), and will not be elaborated here.

[0043] In step 104, cross-modal completion features of the first query modality are obtained based on the similarity scores and SoftMatch weights of the first fine-grained features and the local features of the second query modality.

[0044] Optionally, step 104 includes the following two steps.

[0045] The first step is to calculate the similarity score and SoftMatch weight between each position of the first fine-grained feature and the local features of the second query modality.

[0046] Optionally, the similarity scores between the positions of the first fine-grained feature and the local features of the second query modality are calculated using formula (3).

[0047] (3) In formula (3), This represents the i-th position of the first fine-grained feature. With the j-th position of the second query modality Similarity score between them This indicates the calculation of cosine similarity, where i and j are positive integers, and the value of i ranges from 1 to j. The value of j ranges from 1 to . The total number of dimensions of the first fine-grained feature. This represents the total number of dimensions in the second query modality.

[0048] The second step involves selecting features from the top R positions based on similarity scores, and then weighting and aggregating the features from the top R positions based on the SoftMatch weights of the features to obtain the cross-modal completion features for the first query modality.

[0049] After obtaining the similarity scores between the positions of the first fine-grained feature and the local features of the second query modality, the positions are sorted from largest to smallest according to the similarity scores to select the Top R positions in the sort. Here, R is a fixed percentage, for example, R=50%.

[0050] After normalizing these similarity scores, the SoftMatch weight for each position can be obtained. Then, based on the SoftMatch weight for each position, the Top R positions are weighted and aggregated to obtain the cross-modal completion features for the first query modality. .

[0051] To ensure the first query modality fully integrates complementary information from the second query modality, an alignment loss is introduced during the training phase. This alignment loss is the supervision loss corresponding to the cross-modal completion features of the first query modality. In implementation, mean squared error and cosine distance losses are used to supervise the cross-modal completion features of the first query modality, thus driving the development of the first fine-grained features. Cross-modal completion features with the first query modality Alignment in the feature space. That is, during training, the alignment loss can be designed to ensure that the first fine-grained features are aligned. Cross-modal completion features with the first query modality As the model gradually approaches the target, it learns the implicit association between the first query modality and the second query modality (the cross-modal completion features of the first query modality are essentially the complementary features of the second query modality to the first query modality), thereby improving the final generated first fine-grained features. It can integrate the completion information of the second query modality.

[0052] The alignment loss is expressed by formula (4).

[0053] (4) In formula (4), For alignment loss, This is the cross-modal completion feature for the first query mode. The meanings of the other parameters in formula (4) are the same as those in formula (2), and will not be detailed here.

[0054] In step 105, the cross-modal completion features of the first query modality are residually connected with the global features of the first query modality to obtain the single-modal retrieval features of the first query modality.

[0055] The unimodal retrieval features of the first query modality are used for target re-identification.

[0056] For example, step 105 can be represented by formula (5).

[0057] (5) In formula (5), This is the single-modal retrieval feature for the first query modality. The meanings of the other parameters in formula (5) are the same as those in formulas (1) and (2), and are omitted here in detail.

[0058] In step 106, the single-modal retrieval features of the second query modality are obtained in the same manner as the first query modality.

[0059] When implementing step 106, the same method as in steps 101 to 105 above can be used to obtain the single-modal retrieval features of the second query modality. Simply swap the first query modality with the second query modality; details are omitted here.

[0060] In step 107, the dynamic fusion weights of the first query mode and the second query mode are obtained based on the unimodal retrieval features of the first query mode and the unimodal retrieval features of the second query mode.

[0061] Optionally, the unimodal retrieval features of the first query modality and the unimodal retrieval features of the second query modality are concatenated and input into a multi-layer perceptron (MLP), and then normalized by the Softmax function to obtain the dynamic fusion weights of the first query modality and the second query modality.

[0062] For example, step 107 is represented by formula (6).

[0063] (6) In formula (6), The dynamic fusion weights for the first query modality. For the dynamic fusion weights of the second query modality, This is a single-modal retrieval feature for the second query modality. This indicates that a and b are concatenated. The meanings of the other parameters in formula (6) are the same as those in formula (5), and will not be elaborated here.

[0064] In step 108, based on dynamic fusion weights, the single-modal retrieval features of the first query modality and the single-modal retrieval features of the second query modality are weighted and aggregated to obtain multimodal joint features.

[0065] Multimodal joint features are used for multimodal target re-identification.

[0066] For example, step 108 is represented by formula (7).

[0067] (7) In formula (7), This represents a multimodal joint feature. The meanings of the other parameters in formula (7) are the same as those in formulas (5) and (6), and will not be elaborated here.

[0068] This fusion method can automatically adjust the contribution of each modal feature to the final joint representation based on the effective information content of different instance inputs, achieving more refined information fusion and effectively improving the discrimination performance and robustness in multimodal joint retrieval scenarios where multiple query modalities are available.

[0069] In multimodal query scenarios (text modality and sketch image modality), target re-identification can be performed using the multimodal joint features in step 108; while in unimodal scenarios, target re-identification can be performed using unimodal retrieval features. Here, for cases with only a single modality input, the multimodal target re-identification model can utilize the cross-modal associations learned during training to automatically infer the potential collaborative features of the main modality in the missing modality, thereby significantly improving the discrimination ability in unimodal scenarios.

[0070] This invention realizes a unified multimodal and multi-task target re-identification model, which eliminates the need to design and train dedicated models for different modalities or different retrieval tasks. It is compatible with multiple modal inputs, including text, sketches, and combinations of text and sketches, and supports multiple query paradigms and tasks, greatly improving the model's versatility and actual deployment efficiency.

[0071] Steps 102 to 104 can fully explore the complementary information between different modalities. Whether in a single-modal or multi-modal input scenario, it can automatically complete and enhance feature representation, significantly improving retrieval accuracy and discrimination ability in complex scenarios such as modality missing or uncertain input.

[0072] Step 107 dynamically assigns different modal fusion weights to each instance and adaptively adjusts feature contributions based on the effectiveness of the actual input content, achieving information balance and complementary advantages, thereby further improving the robustness and accuracy of multimodal joint retrieval.

[0073] In this embodiment, local and global features of two query modalities of the i-th sample are obtained. The i-th sample includes a first query modality and a second query modality, which are one of the text modality and the sketch image modality, respectively, and are different from each other. The cross-attention between the Prompt feature vector and the global features of the first query modality is calculated to obtain the fused features. The Prompt feature vector is a learnable feature vector. The cross-attention and self-attention between the fused features and the local features of the first query modality are calculated to obtain the first fine-grained features of the first query modality. Based on the similarity score between the first fine-grained features and the local features of the second query modality and the SoftMatch weight, the cross-modal completion features of the first query modality are obtained. The cross-modal completion features of the first query modality are residually connected with the global features of the first query modality to obtain the single-modal retrieval features of the first query modality. The single-modal retrieval features of the first query modality are used for target re-identification. The single-modal retrieval features of the first query modality not only contain the information of the first query modality itself, but also explicitly integrate the supplementary information of the second query modality, realizing information complementarity and discrimination enhancement in multimodal scenarios, thereby enabling accurate and effective multimodal target re-identification.

[0074] Figure 3 A flowchart of a multimodal target re-identification model training method provided by an exemplary embodiment of the present invention is shown, which can be executed by a computer device. Figure 4 This is a schematic diagram of the structure of a multimodal target re-identification model.

[0075] See Figure 3-4 The method includes: In step 301, the global features of the target queried modality are obtained.

[0076] Here, the target query modality is the query modality in the i-th sample, and the target query modality corresponds to the first query modality and the second query modality. The global features of the target query modality can also be obtained in the same way as the global features of the sketch image modality (e.g., through an image encoder).

[0077] In step 302, the loss function of the multimodal target re-identification model is constructed.

[0078] See Figure 4The multimodal target re-identification model takes two query modalities as input: a text modality and a sketch image modality. The text modality is processed by a text feature encoder, and the image modality is processed by an image feature encoder. Fine-grained features are then extracted distributedly, and residual connections are used to obtain the corresponding single-modal retrieval features (see steps 101 to 106 above for the methods of obtaining single-modal retrieval features). The two single-modal retrieval features are then weighted and fused to obtain the multimodal joint features (see steps 107 to 108 above for the methods of obtaining multimodal joint features). These multimodal joint features can be used for target re-identification.

[0079] In this embodiment, the loss function includes: a first contrast loss, a second contrast loss, a third contrast loss, a consistency loss, and an alignment loss; The first contrast loss is obtained based on the first InfoNCE contrast loss between the single-modal retrieval features of the first query modality and the global features of the target query modality. The second contrast loss is obtained based on the second InfoNCE contrast loss between the single-modal retrieval features of the second query modality and the global features of the target query modality. The third contrast loss is the third InfoNCE contrast loss between the multimodal joint features and the global features of the target query modality. The consistency loss is the normalized similarity distribution of the single-modal retrieval features of the first query modality and the single-modal retrieval features of the second query modality to the global features of the target query modality. The alignment loss is used to promote the alignment of the cross-modal completion features of the first query modality with the first fine-grained features (i.e., formula (4)).

[0080] The following section explains the various losses within the loss function.

[0081] First, the InfoNCE contrast loss between the features of the x-th modality and the global features of the target queried modality is expressed by formula (8). That is, the first InfoNCE contrast loss, the second InfoNCE contrast loss, and the third InfoNCE contrast loss can all be calculated by formula (8).

[0082] (8) In formula (8), This represents the InfoNCE contrastive loss between the features of the x-th modality and the global features of the target query modality. Here, the features of the x-th modality can be the unimodal retrieval features of the first query modality. Single-modal retrieval features of the second query modality Multimodal joint features . This represents the total number of samples in the current training batch. For the features of the x-th mode of the i-th sample, The global features of the target query modality for the i-th sample. The global features of the target query modality for the k-th sample are the other samples in the current training batch that are different from the i-th sample. It is the temperature coefficient.

[0083] To achieve adaptive training balance across multiple tasks, a confidence-weighted mechanism is introduced for the first and second contrastive losses, instead of directly using the InfoNCE contrastive loss. This method includes the following five steps.

[0084] The first step is to generate a confidence score for the first query modality based on the first InfoNCE contrastive loss.

[0085] For example, the confidence score of the first query modality of the i-th sample is represented by formula (9).

[0086] (9) In formula (9), Let be the confidence score of the first query modality for the i-th sample. The first InfoNCE contrast loss for the first query modality of the i-th sample is calculated using formula (8). Substitute into formula (8) ).

[0087] The second step is to generate a confidence score for the second query modality based on the second InfoNCE contrastive loss.

[0088] For example, the confidence score of the second query modality of the i-th sample is represented by formula (9).

[0089] (10) In formula (10), Let be the confidence score of the second query modality for the i-th sample. The second InfoNCE contrastive loss for the second query modality of the i-th sample is calculated using formula (8) (that is, the loss is calculated using formula (8)). Substitute into formula (8) ).

[0090] The third step is to calculate the weighting coefficients of the first query mode and the second query mode based on the confidence scores of the first query mode and the second query mode.

[0091] For example, the weighting coefficient of the first query modality is calculated using formula (11), and the weighting coefficient of the second query modality is calculated using formula (12).

[0092] (11) (12) In formulas (11) and (12), The weighting coefficients for the first query mode. The weighting coefficients are for the second query mode. The meanings of the other parameters in formulas (11) and (12) are the same as those in formulas (9) and (10), and are omitted here.

[0093] The fourth step is to weight the first InfoNCE contrast loss based on the weighting coefficients of the first query modality to obtain the first contrast loss.

[0094] Optionally, the first contrast loss is calculated using formula (13).

[0095] (13) In formula (13), As the first comparison loss, , This is a hyperparameter. The meanings of the other parameters in formula (13) are the same as those in formulas (9) and (12), and will not be elaborated here.

[0096] Fifth, the second InfoNCE contrast loss is weighted based on the weighting coefficients of the second query modality to obtain the second contrast loss.

[0097] (14) In formula (14), This is the second contrast loss. The meanings of the other parameters in formula (14) are the same as those in formulas (10), (11), and (13), and are omitted here for detailed explanation.

[0098] To further constrain the consistency of the retrieval distribution across different modalities, a consistency loss is introduced. This consistency loss is calculated using KL divergence. In this case, the consistency loss is expressed by formula (15).

[0099] (15) In formula (15), This results in a loss of consistency. The unimodal retrieval features of the first query modality for the i-th sample Global features of the target query modality for the i-th sample The normalized similarity distribution. The single-modal retrieval features of the second query modality for the i-th sample Global features of the target query modality for the i-th sample The normalized similarity distribution. The meanings of the other parameters in formula (15) are the same as those in formula (8), and are omitted here.

[0100] In summary, the loss function of the multimodal target re-identification model can be expressed by formula (16).

[0101] (16) In formula (16), The loss function for the multimodal target re-identification model is... The third contrastive loss, namely the third InfoNCE contrastive loss between the multimodal joint features and the global features of the target query modality, can be achieved by combining the multimodal joint features... Substitute into formula (8) Calculated.

[0102] The above optimization objectives ensure that both single-modal and multi-modal joint queries can achieve accurate alignment with target image features in a unified discrimination space, thereby improving the model's retrieval accuracy and generalization ability in multiple tasks and scenarios.

[0103] In step 303, the i-th sample is used to train a multimodal target re-identification model based on the loss function.

[0104] When training a multimodal target re-identification model based on a loss function, the parameters of the multimodal target re-identification model are optimized with the goal of minimizing the loss function.

[0105] Here, when training the multimodal target re-identification model, in addition to the i-th sample, other samples in the same training batch also need to be input into the multimodal target re-identification model and have their corresponding losses calculated in the same way as the i-th sample, so that the loss function value can be calculated.

[0106] After the multimodal target re-identification model is trained, the query can be input into the multimodal target re-identification model. At this time, the multimodal target re-identification model will extract the corresponding features of the query (such as single-modal retrieval features or multimodal joint features), and then perform similarity matching with the image features of each image in the database. The similarity matching results are sorted using Top K, that is, the K images with the highest similarity are used as the retrieval results, thereby realizing target re-identification under the condition of multimodal input and query modality uncertainty.

[0107] For example, this invention conducts experiments on several publicly available multimodal object re-identification datasets, including Tri-CUHK-PEDES, Tri-ICFG-PEDES, and Tri-RSTPReid. All experiments strictly adhere to the publicly available protocols for dividing the training and test sets to ensure fairness and reproducibility. Evaluation employs a cross-modal object re-identification standard protocol, primarily reporting metrics such as Rank-k (k=1, 5, 10) retrieval accuracy, mean average precision (mAP), and mean inverse negative penalty (mINP), comparing the performance of different methods on multimodal retrieval tasks. To comprehensively analyze the adaptability and robustness of this method under different modal combinations and query tasks, the experimental setup includes the following three scenarios.

[0108] (1) Use only text description as the query modality to retrieve the target from the image library.

[0109] (2) Use only sketch images as the query modality to retrieve the target from the image library.

[0110] (3) Text and sketches are combined as a multimodal query to retrieve the target image library.

[0111] This invention's model uses a pre-trained CLIP-ViT-B / 16 as the visual backbone, extracting features from sketches and RGB images respectively, and employing the CLIP text encoder to embed text descriptions. Each training iteration randomly samples 64 identities from the training set, each identity containing one sketch, one text description, and one RGB image, forming a multimodal, multi-task joint training batch. During training, a multi-task loss and confidence-weighted mechanism are used for optimization. , The hyperparameters were set to 1 and 3.5 respectively.

[0112] To improve the model's generalization ability, visual input is augmented with random horizontal flipping, padding, cropping, and erasing, while text input is augmented with random masks to enhance robustness. The optimizer is Adam, with an initial learning rate of 1e-5, gradually decreasing according to a cosine annealing strategy. The temperature parameter of the comparison loss is used in InfoNCE. The value was set to 0.07. All experiments were conducted on a single NVIDIA RTX 4090 GPU, with 60 training epochs.

[0113] To verify the effectiveness of this invention, it is compared with depth map super-resolution methods in related technologies, which mainly include: (1) CMPM / C: Aichun Zhu, Zijie Wang, Yifeng Li, Xili Wan, Jing Jin,Tian Wang, Fangqiang Hu, and Gang Hua. Dssl: Deep surroundings-personseparation learning for text-based person retrieval. In ACMMM,pages 209–217,2021. (2) TIMAM: Nikolaos Sarafianos, Xiang Xu, and Ioannis A Kakadiaris.Adversarial representation learning for text-to-image matching. In ICCV,pages 5814–5824, 2019. (3) GLAM: Ya Jing, Chenyang Si, Junbo Wang, Wei Wang, Liang Wang, andTieniu Tan. Pose-guided joint global and attentive local matching network fortext-based person search. AAAI, 2020. (4) ViTAA: Zhe Wang, Zhiyuan Fang, Jun Wang, and Yezhou Yang. Vitaa:Visual-textual attributes alignment in person search by natural language. InECCV, pages 402–420. Springer, 2020. (5) DSSL: Aichun Zhu, Zijie Wang, Yifeng Li, Xili Wan, Jing Jin, TianWang, Fangqiang Hu, and Gang Hua. DSSL: Deep surroundings-person separation learning for text-based person retrieval. In ACMMM, pages 209–217, 2021. (6) LBUL+BERT: Zijie Wang, Aichun Zhu, Jingyi Xue, Xili Wan, ChaoLiu, Tian Wang, and Yifeng Li. Look before you leap: Improving text-based person retrieval by learning a consistent cross-modal common manifold. In ACMMM, pages 1984–1992, 2022. (7) CAIBC: Zijie Wang, Aichun Zhu, Jingyi Xue, Xili Wan, Chao Liu, Tian Wang, and Yifeng Li. Caibc: Capturing all-round information beyond color for text-based person retrieval. In ACM MM, pages 5314–5322, 2022. (8) LGUR: Zhiyin Shao, Xinyu Zhang, Meng Fang, Zhifeng Lin, Jian Wang, and Changxing Ding. Learning granularity-unified representations for text-to-image person re-identification. In ACM MM, pages 5566–5574, 2022. (9) IVT: Xiujun Shu, Wei Wen, Haoqian Wu, Keyu Chen, Yiran Song, Ruizhi Qiao, Bo Ren, and Xiao Wang. See finer, see more: Implicit modalityalignment for text-based person retrieval. In ECCV, pages 624–641. Springer, 2022. (10) UNIReID: Cuiqun Chen, Mang Ye, and Ding Jiang. Towards modality-agnostic person re-identification with descriptive query. In CVPR, pages15128–15137, 2023. Eleven target re-identification methods (ten methods in related technologies and the method in this invention) were tested on the Tri-CUHKPEDES, Tri-ICFGPEDES and Tri-RSTPReid datasets, respectively. The test results are shown in Table 1.

[0114] Table 1: Accuracy of the target re-identification method of this invention and 10 related techniques on the Tri-CUHKPEDES, Tri-ICFGPEDES and Tri-RSTPReid datasets.

[0115]

[0116] As shown in Table 1, the unified multimodal multitask recognition model of this invention achieves state-of-the-art comprehensive performance in various practical application scenarios, including text-based image retrieval, sketch-based image retrieval, and joint text-and-sketch image retrieval. This result fully demonstrates the effectiveness of the cross-modal interactive completion mechanism and information-adaptive fusion strategy proposed in this invention. Through a unified model structure and end-to-end multitask optimization, this method can adaptively handle query tasks with different modal inputs, breaking through the information expression bottleneck of previous single-modal or simple splicing methods, and better mining and fusing complementary information from various modalities. Experimental results show that this invention can maintain its leading advantage when facing different tasks, different modal combinations, and even modal uncertainty, proving the significant progress of the technical solution in terms of generalization, robustness, and practical application. The table results confirm the innovation and practical application value of this invention from multiple dimensions.

[0117] Table 2: Results of module ablation experiments conducted on the Tri-CUHKPEDES dataset according to this invention.

[0118]

[0119] In Table 2, the cross-modal completion module corresponds to steps 102 to 104, and the information content adaptive fusion module corresponds to step 107. Ablation experiments on Tri-CUHK-PEDES show that the cross-modal completion module and the information content adaptive fusion module each play key roles and are significantly complementary within the framework of this invention. The cross-modal completion module improves the single-modal retrieval performance by strengthening semantic expression, resulting in a significant improvement in Rank-1 for both text retrieval and sketch retrieval. The information content adaptive fusion module, used alone, improves Rank-1 and mAP in multimodal joint retrieval, indicating that it can effectively alleviate the fusion bias caused by inconsistent modal information content. When both are used simultaneously, the model achieves the highest Rank-1, mAP, and mINP, verifying that the semantic supplementation capability of the cross-modal completion module and the adaptive fusion mechanism of the information content adaptive fusion module mutually enhance each other, thus jointly bringing optimal performance.

[0120] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the above method embodiments.

[0121] Figure 5 A schematic diagram of the structure of a multimodal target re-identification device provided in an exemplary embodiment of the present invention is shown. See also Figure 5 The multimodal target re-identification device 500 includes: an acquisition module 501, a first attention module 502, a second attention module 503, a cross-modal completion module 504, a residual connection module 505, and a multimodal joint module 506.

[0122] The acquisition module 501 is used to acquire the local and global features of the two query modalities of the i-th sample. The i-th sample includes a first query modality and a second query modality. The first query modality and the second query modality are one of the text modality and the sketch image modality, respectively. The first query modality and the second query modality are different. The first attention module 502 is used to calculate the cross attention between the Prompt feature vector and the global features of the first query modality to obtain the fused features. The Prompt feature vector is a learnable feature vector. The second attention module 503 is used to calculate the cross attention and self attention between the fused features and the local features of the first query modality to obtain the first fine-grained features of the first query modality; The cross-modal completion module 504 is used to obtain the cross-modal completion features of the first query modality based on the similarity score and SoftMatch weight of the first fine-grained features and the local features of the second query modality. The residual connection module 505 is used to perform a residual connection between the cross-modal completion features of the first query modality and the global features of the first query modality to obtain the single-modal retrieval features of the first query modality. The single-modal retrieval features of the first query modality are used for target re-identification.

[0123] Optionally, the cross-modal completion module 504 is also used to calculate the similarity score and SoftMatch weight between each position of the first fine-grained feature and the local feature of the second query modality; based on the similarity score, the features of the Top R positions are selected, and the features of the Top R positions are weighted and aggregated based on the SoftMatch weight of the features of the Top R positions to obtain the cross-modal completion features of the first query modality.

[0124] Optionally, the multimodal joint module 506 is used to obtain the single-modal retrieval features of the second query modality in the same way as the first query modality; obtain the dynamic fusion weights of the first query modality and the second query modality based on the single-modal retrieval features of the first query modality and the second query modality; and perform weighted aggregation on the single-modal retrieval features of the first query modality and the second query modality based on the dynamic fusion weights to obtain multimodal joint features, which are used for multimodal target re-identification.

[0125] Figure 6 A schematic diagram of a multimodal target re-identification model training device provided in an exemplary embodiment of the present invention is shown. See also Figure 6 The multimodal target re-identification model training device 600 includes: an acquisition module 601, a loss function construction module 602, and a training module 603.

[0126] The acquisition module 601 is used to acquire the global features of the target queried modality; The loss function construction module 602 is used to construct the loss function of the multimodal target re-identification model; Training module 603 is used to train a multimodal target re-identification model based on a loss function using the i-th sample; The loss functions include: first contrast loss, second contrast loss, third contrast loss, consistency loss, and alignment loss; The first contrast loss is obtained based on the first InfoNCE contrast loss between the single-modal retrieval features of the first query modality and the global features of the target query modality. The second contrast loss is obtained based on the second InfoNCE contrast loss between the single-modal retrieval features of the second query modality and the global features of the target query modality. The third contrast loss is the third InfoNCE contrast loss between the multimodal joint features and the global features of the target query modality. The consistency loss is the normalized similarity distribution of the single-modal retrieval features of the first query modality and the single-modal retrieval features of the second query modality to the global features of the target query modality. The alignment loss is used to promote the alignment of the cross-modal completion features of the first query modality with the first fine-grained features. Among them, the single-modal retrieval features of the first query modality, the single-modal retrieval features of the second query modality, the multimodal joint features, the cross-modal completion features of the first query modality, and the first fine-grained features are obtained using a multimodal target re-identification method.

[0127] In the loss function construction module 602, the first contrastive loss and the second contrastive loss are obtained as follows: based on the first InfoNCE contrastive loss, a confidence score for the first query mode is generated; based on the second InfoNCE contrastive loss, a confidence score for the second query mode is generated; based on the confidence scores of the first and second query modes, weighting coefficients for the first and second query modes are calculated; the first InfoNCE contrastive loss is weighted based on the weighting coefficients of the first query mode to obtain the first contrastive loss; the second InfoNCE contrastive loss is weighted based on the weighting coefficients of the second query mode to obtain the second contrastive loss.

[0128] It should be noted that the multimodal target re-identification device provided in the above embodiments, when performing multimodal target re-identification, or the multimodal target re-identification model training device provided in the above embodiments, when training a multimodal target re-identification model, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the multimodal target re-identification device and the multimodal target re-identification method embodiments provided in the above embodiments belong to the same concept, and the multimodal target re-identification model training device and the multimodal target re-identification model training method embodiments provided in the above embodiments belong to the same concept. The specific implementation process is detailed in the method embodiments and will not be repeated here.

[0129] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in each embodiment of the invention can be integrated into a single processor, exist as separate physical entities, or consist of two or more modules integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0130] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a personal computer, mobile phone, or communication device, etc.) or processor to execute all or part of the steps of the method of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] Figure 7 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of the present invention. For example... Figure 7 As shown, the computer device 700 includes a processor 701 and a memory 702.

[0132] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0133] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 is used to store at least one instruction, which is executed by the processor 701 to implement the multimodal target re-identification method and the multimodal target re-identification model training method provided in the embodiments of the present invention.

[0134] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the computer device 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0135] This invention also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of a computer device, the computer device is able to execute the multimodal target re-identification method and the multimodal target re-identification model training method provided in this invention.

[0136] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the multimodal target re-identification method and the multimodal target re-identification model training method provided in this invention.

[0137] The above description is merely an optional embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal target re-identification method, characterized in that, The method includes: Obtain the local and global features of the two query modalities of the i-th sample. The i-th sample includes a first query modality and a second query modality. The first query modality and the second query modality are one of the text modality and the sketch image modality, respectively. The first query modality and the second query modality are different. The cross-attention between the Prompt feature vector and the global features of the first query modality is calculated to obtain the fused features, wherein the Prompt feature vector is a learnable feature vector; Calculate the cross-attention and self-attention between the fused features and the local features of the first query modality to obtain the first fine-grained features of the first query modality; Based on the similarity score and SoftMatch weight between the first fine-grained feature and the local features of the second query modality, the cross-modal completion feature of the first query modality is obtained; The cross-modal completion features of the first query modality are residually connected with the global features of the first query modality to obtain the single-modal retrieval features of the first query modality. The single-modal retrieval features of the first query modality are used for target re-identification.

2. The method according to claim 1, characterized in that, The step of obtaining cross-modal completion features of the first query modality based on the similarity score and SoftMatch weight of the first fine-grained feature and the local features of the second query modality includes: Calculate the similarity score and SoftMatch weight between each position of the first fine-grained feature and the local features of the second query modality; Based on the similarity scores, the features of the Top R positions are selected, and the features of the Top R positions are weighted and aggregated based on the SoftMatch weights of the features of the Top R positions to obtain the cross-modal completion features of the first query modality.

3. The method according to claim 1, characterized in that, The method further includes: The single-modal retrieval features of the second query modality are obtained using the same method as those of the first query modality. Based on the single-modal retrieval features of the first query modality and the single-modal retrieval features of the second query modality, obtain the dynamic fusion weights of the first query modality and the second query modality; Based on the dynamic fusion weights, the single-modal retrieval features of the first query modality and the single-modal retrieval features of the second query modality are weighted and aggregated to obtain multimodal joint features, which are used for multimodal target re-identification.

4. A method for training a multimodal target re-identification model, characterized in that, The i-th sample includes the target query modality, wherein the target query modality is a visible light image modality, and the method includes: Obtain the global features of the target queried modality; Construct the loss function for the multimodal target re-identification model; Using the i-th sample, the multimodal target re-identification model is trained based on the loss function; The loss function includes: first contrast loss, second contrast loss, third contrast loss, consistency loss, and alignment loss; The first contrast loss is obtained based on the first InfoNCE contrast loss between the unimodal retrieval features of the first query modality and the global features of the target query modality. The second contrast loss is obtained based on the second InfoNCE contrast loss between the unimodal retrieval features of the second query modality and the global features of the target query modality. The third contrast loss is the third InfoNCE contrast loss between the multimodal joint features and the global features of the target query modality. The consistency loss is the normalized similarity distribution of the unimodal retrieval features of the first query modality and the unimodal retrieval features of the second query modality with respect to the global features of the target query modality. The alignment loss is used to promote the alignment of the cross-modal completion features of the first query modality with the first fine-grained features. The single-modal retrieval features of the first query modality, the single-modal retrieval features of the second query modality, the multimodal joint features, the cross-modal completion features of the first query modality, and the first fine-grained features are obtained using the method described in claim 3.

5. The method according to claim 4, characterized in that, The first contrast loss and the second contrast loss are obtained in the following manner: Based on the first InfoNCE contrastive loss, a confidence score for the first query modality is generated; Based on the second InfoNCE contrastive loss, a confidence score for the second query modality is generated; Based on the confidence scores of the first query modality and the second query modality, calculate the weighting coefficients of the first query modality and the second query modality; The first InfoNCE contrast loss is obtained by weighting the first query modality with the weighting coefficients. The second contrast loss is obtained by weighting the second InfoNCE contrast loss based on the weighting coefficient of the second query modality.

6. A multimodal target re-identification device, characterized in that, The device includes: The acquisition module is used to acquire the local and global features of the two query modalities of the i-th sample. The i-th sample includes a first query modality and a second query modality. The first query modality and the second query modality are one of the text modality and the sketch image modality, respectively. The first query modality and the second query modality are different. The first attention module is used to calculate the cross-attention between the Prompt feature vector and the global features of the first query modality to obtain fused features. The Prompt feature vector is a learnable feature vector. The second attention module is used to calculate the cross attention and self attention between the fused features and the local features of the first query modality to obtain the first fine-grained features of the first query modality. The cross-modal completion module is used to obtain the cross-modal completion features of the first query modality based on the similarity score and SoftMatch weight between the first fine-grained features and the local features of the second query modality; The residual connection module is used to perform a residual connection between the cross-modal completion features of the first query modality and the global features of the first query modality to obtain the single-modal retrieval features of the first query modality. The single-modal retrieval features of the first query modality are used for target re-identification.

7. A multimodal target re-identification model training device, characterized in that, The i-th sample includes the target query modality, wherein the target query modality is a visible light image modality, and the device includes: The acquisition module is used to acquire the global features of the target queried modality; The loss function construction module is used to construct the loss function for the multimodal target re-identification model; The training module is used to train the multimodal target re-identification model based on the loss function using the i-th sample; The loss function includes: first contrast loss, second contrast loss, third contrast loss, consistency loss, and alignment loss; The first contrast loss is obtained based on the first InfoNCE contrast loss between the unimodal retrieval features of the first query modality and the global features of the target query modality. The second contrast loss is obtained based on the second InfoNCE contrast loss between the unimodal retrieval features of the second query modality and the global features of the target query modality. The third contrast loss is the third InfoNCE contrast loss between the multimodal joint features and the global features of the target query modality. The consistency loss is the normalized similarity distribution of the unimodal retrieval features of the first query modality and the unimodal retrieval features of the second query modality with respect to the global features of the target query modality. The alignment loss is used to promote the alignment of the cross-modal completion features of the first query modality with the first fine-grained features. The single-modal retrieval features of the first query modality, the single-modal retrieval features of the second query modality, the multimodal joint features, the cross-modal completion features of the first query modality, and the first fine-grained features are obtained using the method described in claim 3.

8. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the method of any one of claims 1 to 3 or claims 4 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the method of any one of claims 1 to 3 or claims 4 to 5.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1 to 3 or claims 4 to 5.

Images