Multi-modal target re-identification method and related equipment

Abstract

The invention discloses a multi-modal target re-identification method and related equipment, and the method comprises the steps: generating a global text feature according to a multi-modal image based on a multi-modal unified text generation strategy; generating a multi-modal image token according to the multi-modal image through a segmentation guide feature modulation module; fusing the multi-modal semantic query token and the global text features through a semantic token redistribution module to generate a multi-modal query sequence; performing cross attention reconstruction through a semantic token redistribution module according to the multi-modal query sequence and the multi-modal image tokens to obtain a cross-modal semantic token set; performing cross-modal hypergraph interaction operation according to the cross-modal semantic token set and the global image features of the multi-modal image through a cross-modal hypergraph interaction module to generate global fusion features; the global fusion feature is used for completing re-identification of the to-be-identified target. The identity matching accuracy in a complex scene can be remarkably improved, and the method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a multimodal target re-identification method and related equipment. Background Technology

[0002] Multimodal object re-identification (ReID) technology aims to overcome the limitations of single-modality methods in complex scenarios such as varying lighting conditions, nighttime, or inclement weather by fusing multiple spectral information sources (e.g., visible light (RGB), near-infrared (NIR), and thermal infrared (TIR)). This enhances the robustness and accuracy of the recognition system. Compared to traditional single-modal ReID methods, multimodal re-identification fully utilizes the complementary characteristics between different modalities, exhibiting stronger environmental adaptability and becoming a current research hotspot. However, in practical applications, related technologies still face challenges such as information loss due to token selection, insufficient cross-modal semantic relationship modeling capabilities, and limitations in multimodal text generation.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The embodiments of this application aim to at least partially address one of the technical problems in the related art. Therefore, the main objective of the embodiments of this application is to propose a multimodal target re-identification method and related equipment, which can significantly improve the accuracy of identity matching in complex scenarios.

[0005] To achieve the above objectives, one aspect of this application proposes a multimodal target re-identification method, the method comprising the following steps: Acquire multimodal images of the target to be identified; wherein the multimodal images include visible light images, near-infrared images, and thermal infrared images; Based on the multimodal unified text generation strategy, global text features are generated from the multimodal images; The segmentation-guided feature modulation module generates a multimodal image token based on the multimodal image; The multimodal semantic query token is fused with the global text features by the semantic token reassignment module to generate a multimodal query sequence; The semantic token redistribution module performs cross-attention reconstruction based on the multimodal query sequence and the multimodal image tokens to obtain a cross-modal semantic token set; The cross-modal hypergraph interaction module performs cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of the multimodal image to generate global fusion features; the global fusion features are used to complete the re-identification of the target to be identified.

[0006] In some embodiments, generating global text features based on the multimodal unified text generation strategy according to the multimodal image includes: The visible light image, near-infrared image, and thermal infrared image in the multimodal image are horizontally stitched together to generate a composite image; The visible light image, the near-infrared image, the thermal infrared image, and the composite image are respectively input into a multimodal large language model, and the semantic attribute prediction results corresponding to the visible light image, the near-infrared image, the thermal infrared image, and the composite image are output. Among them, each image input path is designed with a target prompt word to guide the multimodal large language model to focus on the characteristics of different modalities based on the target prompt word. The corresponding target attribute values ​​are selected from the semantic attribute prediction results corresponding to the visible light image, the near-infrared image, the thermal infrared image, and the composite image, respectively. The target attribute values ​​corresponding to the visible light image, the near-infrared image, the thermal infrared image, and the composite image are input into a predefined structured template to generate global text description information; The global text description information is encoded by a text encoder to obtain the global text features.

[0007] In some embodiments, the method further includes: The visible light image, the near-infrared image, and the thermal infrared image are respectively masked and segmented using the pre-trained segmentation model in the segmentation-guided feature modulation module to obtain a binary foreground segmentation mask set; The visible light image, the near-infrared image, and the thermal infrared image are segmented respectively to obtain image blocks corresponding to the visible light image, the near-infrared image, and the thermal infrared image; Based on the image patches of the visible light image and the binary foreground segmentation mask of the visible light image, a binary mask vector of the visible light image is generated; Based on the image patches of the near-infrared image and the binary foreground segmentation mask of the near-infrared image, a binary mask vector of the near-infrared image is generated; Based on the image blocks of the thermal infrared image and the binary foreground segmentation mask of the thermal infrared image, a binary mask vector of the thermal infrared image is generated. Based on the original attention from the attention modulation mechanism in the segmentation-guided feature modulation module, a positive modulation matrix and a negative modulation matrix are calculated; wherein, the positive modulation matrix is ​​used to enhance foreground interaction, and the negative modulation matrix is ​​used to suppress background interaction; Based on the binary mask vectors corresponding to the visible light image, the near-infrared image, and the thermal infrared image, a corresponding foreground interaction guidance matrix is ​​constructed; Based on the positive modulation matrix and the negative modulation matrix, and combined with the foreground interaction guidance matrices corresponding to the visible light image, the near-infrared image, and the thermal infrared image, the attention weights corresponding to the visible light image, the near-infrared image, and the thermal infrared image are calculated; wherein, the attention weights are used to guide the model's attention to the target subject.

[0008] In some embodiments, the multimodal semantic query token includes a semantic query token corresponding to the visible light image, a semantic query token corresponding to the near-infrared image, and a semantic query token corresponding to the thermal infrared image. The step of fusing the multimodal semantic query token with the global text features through a semantic token reassignment module to generate a multimodal query sequence includes: The global text features are concatenated to the end of the sequence of the semantic query token corresponding to the visible light image to generate the first query sequence corresponding to the visible light image; The global text features are concatenated to the end of the sequence of the semantic query token corresponding to the near-infrared image to generate the second query sequence corresponding to the near-infrared image; The global text features are concatenated to the end of the sequence of the semantic query token corresponding to the thermal infrared image to generate the third query sequence corresponding to the thermal infrared image; The multimodal query sequence is constructed based on the first query sequence, the second query sequence, and the third query sequence.

[0009] In some embodiments, the step of obtaining a cross-modal semantic token set by performing cross-attention reconstruction based on the multimodal query sequence and the multimodal image tokens through the semantic token redistribution module includes: A first cross-attention calculation is performed based on the first query sequence corresponding to the visible light image contained in the multimodal query sequence and the first image token corresponding to the visible light image contained in the multimodal image token. The result of the first cross-attention calculation is then semantically enhanced using a feedforward network to obtain the first target semantic token corresponding to the visible light image. A second cross-attention calculation is performed based on the second query sequence corresponding to the near-infrared image contained in the multimodal query sequence and the second image token corresponding to the near-infrared image contained in the multimodal image token. The result of the second cross-attention calculation is then semantically enhanced using a feedforward network to obtain the second target semantic token corresponding to the near-infrared image. A third cross-attention calculation is performed based on the third query sequence corresponding to the thermal infrared image contained in the multimodal query sequence and the third image token corresponding to the thermal infrared image contained in the multimodal image token. The result of the third cross-attention calculation is then semantically enhanced by combining a feedforward network to obtain the third target semantic token corresponding to the thermal infrared image. The cross-modal semantic token set is constructed based on the first target semantic token, the second target semantic token, and the third target semantic token.

[0010] In some embodiments, the step of performing cross-modal hypergraph interaction operations through the cross-modal hypergraph interaction module based on the cross-modal semantic token set and the global image features of the multimodal image to generate global fusion features includes: The cross-modal semantic token set is determined as the vertex set of the hypergraph; Calculate the semantic similarity matrix among all vertices in the vertex set, and dynamically generate a hyperedge set based on the semantic similarity matrix and a preset similarity threshold; Construct the hypergraph based on the vertex set and the hyperedge set; A hypergraph convolutional network is used to propagate information on the constructed hypergraph to obtain a set of hypergraph node information; Residual connections and feature fusion are performed on the hypergraph node information set to obtain a fused semantic token; The global image features of the fused semantic token and the multimodal image are aggregated to obtain the global fused features.

[0011] To achieve the above objectives, another aspect of this application proposes a multimodal target re-identification device, which includes the following modules: A multimodal image acquisition module is used to acquire multimodal images of the target to be identified; wherein the multimodal images include visible light images, near-infrared images, and thermal infrared images; A global text feature generation module is used to generate global text features based on the multimodal unified text generation strategy and the multimodal image. A multimodal image token generation module is used to generate a multimodal image token based on the multimodal image through a segmentation-guided feature modulation module; The multimodal query sequence generation module is used to fuse the multimodal semantic query token with the global text features through the semantic token redistribution module to generate a multimodal query sequence. The cross-attention reconstruction module is used to perform cross-attention reconstruction based on the multimodal query sequence and the multimodal image tokens through the semantic token redistribution module to obtain a cross-modal semantic token set; The hypergraph interaction module is used to perform cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of the multimodal image, and generate global fusion features; the global fusion features are used to complete the re-identification of the target to be identified.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0014] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides a multimodal target re-identification method and related equipment. This scheme acquires multimodal images of the target to be identified; wherein, the multimodal images include visible light images, near-infrared images, and thermal infrared images; based on a multimodal unified text generation strategy, global text features are generated according to the multimodal images; a segmentation-guided feature modulation module generates multimodal image tokens according to the multimodal images; a semantic token redistribution module fuses the multimodal semantic query tokens with the global text features to generate a multimodal query sequence; a semantic token redistribution module performs cross-attention reconstruction based on the multimodal query sequence and the multimodal image tokens to obtain a cross-modal semantic token set; a cross-modal hypergraph interaction module performs cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of the multimodal images to generate global fusion features; the global fusion features are used to complete the re-identification of the target to be identified. This application's embodiments fuse image information from all modalities through a multimodal unified text generation strategy to generate a single, structured natural language description, avoiding the bias of a single modality, improving the quality of semantic information, and optimizing the computation process. A semantic token redistribution module fuses multimodal semantic query tokens with global text features and performs cross-attention reconstruction based on multimodal query sequences and multimodal image tokens, generating a more compact and discriminative feature representation while ensuring semantic integrity, thus solving the semantic loss and feature confusion problems caused by current token selection methods. A cross-modal hypergraph interaction module performs cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of multimodal images, breaking through the limitations of traditional pairwise interactions. This allows for the capture and utilization of deeper and more complex complementary information between modalities, significantly enhancing the expressive power of features, especially demonstrating stronger performance in handling challenges such as occlusion and local blurring, and significantly improving the accuracy of identity matching in complex scenarios. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of a multimodal target re-identification method provided in an embodiment of this application; Figure 2 yes Figure 1 The flowchart of step S102 in the text; Figure 3 This is a flowchart illustrating the calculation steps of an adaptive reallocation mechanism provided in an embodiment of this application; Figure 4 yes Figure 1 The flowchart of step S104 in the text; Figure 5 yes Figure 1 The flowchart of step S105 in the text; Figure 6 yes Figure 1The flowchart of step S106 in the text; Figure 7 This is a system framework diagram of a multimodal target re-identification method provided in an embodiment of this application; Figure 8 This is a vehicle visualization heatmap on a dataset, representing a multimodal target re-identification method provided in this application embodiment; Figure 9 This is a pedestrian visualization heatmap on a dataset, representing a multimodal target re-identification method provided in this application embodiment; Figure 10 This is a schematic diagram of the structure of a multimodal target re-identification device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0019] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0022] (1) ReID (Person / Vehicle Re-Identification) refers to the technology of matching and retrieving the identity of the same pedestrian or vehicle under surveillance cameras at different times and locations.

[0023] (2) Multi-modal, in the embodiments of this application, refers to the joint analysis of data from multiple different types of sensors, such as visible light (RGB), near infrared (NIR), thermal infrared (TIR), etc.

[0024] Multimodal object re-identification (ReID) technology aims to overcome the limitations of single-modality methods in complex scenarios such as varying lighting conditions, nighttime conditions, or inclement weather by fusing multiple spectral information sources (e.g., visible light (RGB), near-infrared (NIR), and thermal infrared (TIR)). This enhances the robustness and accuracy of the identification system. Compared to traditional single-modal ReID methods, multimodal re-identification fully utilizes the complementary characteristics between different modalities, exhibiting stronger environmental adaptability and has become a current research hotspot.

[0025] However, the relevant technologies still face the following three key problems in practical applications: (1) Information loss due to token selection: In order to improve model efficiency or focus on key target regions, some related methods adopt token selection or token pruning strategies to actively delete regions in the image that are judged to be "redundant". Although this "hard" selection mechanism helps to reduce computational complexity, it will also irreversibly discard potentially useful information. Especially in the case of occlusion, complex background, etc., the accidental deletion of important regions will directly lead to the loss of semantic information, which will seriously affect the model's discrimination ability and generalization performance.

[0026] (2) Insufficient ability to model cross-modal semantic relationships: Most current multimodal fusion methods only model pairwise or low-order associations between modalities, making it difficult to effectively capture the complex and high-order semantic structures between cross-modal features. For example, a certain detailed region in a visible light image may have a nonlinear correspondence with multiple hotspot regions in a thermal infrared image. Such high-order semantic dependencies are difficult to accurately model using traditional attention mechanisms or simple stitching strategies, thus limiting the deep fusion and complementary utilization of multimodal information.

[0027] (3) Limitations of Multimodal Text Generation: With the development of Multimodal Large Language Models (MLLM), some studies have attempted to introduce text descriptions to assist visual feature modeling in order to enhance the semantic understanding ability of the model. However, current multimodal text generation methods still have the following obvious defects: 1) Modal inconsistency: Most methods generate descriptions based only on RGB modalities, ignoring the supplementary information provided by NIR and TIR, resulting in incomplete or even significantly biased generated text; 2) Semantic ambiguity: Under challenging conditions such as low light, occlusion, or image blur, MLLM often has difficulty accurately extracting target attributes and is prone to generating uncertain words such as "unknown" and "fuzzy", which reduces semantic quality; 3) Low computational efficiency: Some methods require generating text descriptions separately for each modality, resulting in multiple calls to the language model and encoder, significantly increasing computational costs and limiting their deployment and application in real-world scenarios.

[0028] In view of this, this application provides a multimodal target re-identification method and related device. This method acquires multimodal images of the target to be identified, including visible light images, near-infrared images, and thermal infrared images. Based on a multimodal unified text generation strategy, global text features are generated from the multimodal images. A segmentation-guided feature modulation module generates multimodal image tokens from the multimodal images. A semantic token redistribution module fuses the multimodal semantic query tokens with the global text features to generate a multimodal query sequence. The semantic token redistribution module performs cross-attention reconstruction based on the multimodal query sequence and the multimodal image tokens to obtain a cross-modal semantic token set. A cross-modal hypergraph interaction module performs cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of the multimodal images to generate global fusion features. These global fusion features are used to complete the re-identification of the target. This application's embodiments fuse image information from all modalities through a multimodal unified text generation strategy to generate a single, structured natural language description, avoiding the bias of a single modality, improving the quality of semantic information, and optimizing the computation process. A semantic token redistribution module fuses multimodal semantic query tokens with global text features and performs cross-attention reconstruction based on multimodal query sequences and multimodal image tokens, generating a more compact and discriminative feature representation while ensuring semantic integrity, thus solving the semantic loss and feature confusion problems caused by current token selection methods. A cross-modal hypergraph interaction module performs cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of multimodal images, breaking through the limitations of traditional pairwise interactions. This allows for the capture and utilization of deeper and more complex complementary information between modalities, significantly enhancing the expressive power of features, especially demonstrating stronger performance in handling challenges such as occlusion and local blurring, and significantly improving the accuracy of identity matching in complex scenarios.

[0029] This application provides a multimodal target re-identification method, relating to the field of computer technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a multimodal target re-identification method, but is not limited to the above forms.

[0030] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0031] Please see Figure 1 , Figure 1 This is an optional flowchart of a multimodal target re-identification method provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0032] Step S101: Obtain a multimodal image of the target to be identified; wherein the multimodal image includes a visible light image, a near-infrared image, and a thermal infrared image; Optionally, the types of targets to be identified are mainly divided into pedestrian targets and non-pedestrian targets. Non-pedestrian targets may include, but are not limited to, vehicle targets.

[0033] In the embodiments of this application, multimodal images include visible light images (RGB), near-infrared images (NIR), and thermal infrared images (TIR).

[0034] Step S102: Based on the multimodal unified text generation strategy, generate global text features according to the multimodal image; Please see Figure 2 , Figure 2 yes Figure 1 The flowchart of step S102 in the text is as follows: Figure 2 As shown, in some embodiments, step S102 may include, but is not limited to, steps S201 to S205: Step S201: Horizontally stitch together the visible light image, the near-infrared image, and the thermal infrared image in the multimodal image to generate a composite image; Step S202: Input the visible light image, the near-infrared image, the thermal infrared image, and the composite image into the multimodal large language model respectively, and output the semantic attribute prediction results corresponding to the visible light image, the near-infrared image, the thermal infrared image, and the composite image; wherein, each image input path is designed with a target prompt word to guide the multimodal large language model to focus on the characteristics of different modalities according to the target prompt word; Step S203: Select the corresponding target attribute value from the semantic attribute prediction results corresponding to the visible light image, the near-infrared image, the thermal infrared image, and the composite image, respectively. Step S204: Input the target attribute values ​​corresponding to the visible light image, the near-infrared image, the thermal infrared image, and the composite image into a predefined structured template to generate global text description information; Step S205: Encode the global text description information using a text encoder to obtain the global text features.

[0035] In its implementation, this application first proposes a unified text generation strategy to address the problems of modal inconsistency, semantic ambiguity, and low computational efficiency in related methods. The specific implementation process is as follows: 1) Multipath Image Input: For an identity sample, the system horizontally stitches together three images of the same target: a visible light image (RGB), a near-infrared image (NIR), and a thermal infrared image (TIR), to form a composite image. At the same time, the original three single-modal images are retained.

[0036] 2) Fine-grained attribute extraction: The four images (three unimodal images and one composite image) are each input into a Multi-Modal Large Language Model (MLLM). Specific prompts are designed for each input path to guide the model to focus on characteristics of different modalities (such as color, texture, and thermal radiation). The MLLM outputs a series of "attribute-value-confidence" triples (i.e., semantic attribute prediction results) for each input image.

[0037] 3) Structured text fusion: For each semantic attribute (such as "clothing color" or "whether a backpack is carried"), the system gathers all prediction results from the four input paths and selects the value with the highest confidence to fill in the data. Finally, these optimal attribute values ​​(i.e., target attribute values) are filled into a predefined structured template to generate a single, high-confidence, and complete final text description (i.e., global text description information).

[0038] 4) Text Encoding: The generated uniform text (i.e. the final text description) is encoded into a global text feature vector T using CLIP’s text encoder. The global text feature vector can be simply referred to as global text feature.

[0039] In step S102, a single, structured natural language description is generated by fusing image information from all modalities. During this process, a confidence-aware imputation strategy is used to supplement the missing information of low-confidence modalities using attributes of high-confidence modalities. This fundamentally solves three major pain points of related text generation methods: 1) Modality consistency: The description integrates complementary information from all modalities, avoiding the bias of a single modality; 2) Semantic clarity: Confidence imputation significantly reduces uncertain words such as "unknown" and "fuzzy," generating a more reliable and information-rich description; 3) Computational efficiency: Only one text is generated and encoded, avoiding computational redundancy caused by multiple text descriptions and significantly improving training and inference efficiency.

[0040] Step S103: The segmentation-guided feature modulation module generates a multimodal image token based on the multimodal image; This application also proposes an attention modulation mechanism based on segmentation masks, which utilizes prior segmentation knowledge to accurately guide the model to enhance foreground features and suppress background noise, avoiding information loss caused by hard token selection. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a flowchart of the calculation steps for an adaptive reallocation mechanism provided in an embodiment of this application, as follows: Figure 3 As shown, in some embodiments, the calculation process of the attention modulation mechanism based on segmentation mask may include, but is not limited to, steps S301 to S308: Step S301: The visible light image, the near-infrared image, and the thermal infrared image are respectively masked and segmented using the pre-trained segmentation model in the segmentation guidance feature modulation module to obtain a binary foreground segmentation mask set; Step S302: Perform image segmentation on the visible light image, the near-infrared image, and the thermal infrared image respectively to obtain image blocks corresponding to the visible light image, the near-infrared image, and the thermal infrared image; Step S303: Generate a binary mask vector for the visible light image based on the image blocks of the visible light image and the binary foreground segmentation mask of the visible light image; Step S304: Generate a binary mask vector for the near-infrared image based on the image blocks of the near-infrared image and the binary foreground segmentation mask of the near-infrared image; Step S305: Generate a binary mask vector for the thermal infrared image based on the image blocks of the thermal infrared image and the binary foreground segmentation mask of the thermal infrared image. Step S306: Based on the original attention of the attention modulation mechanism in the segmentation guidance feature modulation module, calculate the positive modulation matrix and the negative modulation matrix; wherein, the positive modulation matrix is ​​used to enhance foreground interaction, and the negative modulation matrix is ​​used to suppress background interaction; Step S307: Construct a corresponding foreground interaction guidance matrix based on the binary mask vectors corresponding to the visible light image, the near-infrared image, and the thermal infrared image; Step S308: Based on the positive modulation matrix and the negative modulation matrix, and combined with the foreground interaction guidance matrix corresponding to the visible light image, the near-infrared image, and the thermal infrared image, calculate the attention weights corresponding to the visible light image, the near-infrared image, and the thermal infrared image; wherein, the attention weights are used to guide the model's attention to the target subject.

[0041] In this embodiment, a mask generated by a pre-trained segmentation model is used as prior knowledge to guide the attention mechanism in weighting multimodal image tokens, thereby enhancing the foreground region and suppressing background noise. Compared to methods that directly use segmentation maps as auxiliary input or perform hard cropping, this embodiment deeply integrates segmentation information into the attention calculation, achieving more refined and consistent feature enhancement and effectively improving the model's robustness in complex backgrounds.

[0042] Step S104: The multimodal semantic query token is fused with the global text features through the semantic token redistribution module to generate a multimodal query sequence; Optionally, the multimodal semantic query token includes a semantic query token corresponding to a visible light image, a semantic query token corresponding to a near-infrared image, and a semantic query token corresponding to a thermal infrared image.

[0043] Please see Figure 4 , Figure 4 yes Figure 1 The flowchart of step S104 in the example is as follows: Figure 4 As shown, in some embodiments, step S104 may include, but is not limited to, steps S401 to S404: Step S401: Concatenate the global text features to the end of the sequence of the semantic query token corresponding to the visible light image to generate the first query sequence corresponding to the visible light image; Step S402: The global text features are concatenated to the end of the sequence of the semantic query token corresponding to the near-infrared image to generate the second query sequence corresponding to the near-infrared image; Step S403: The global text features are concatenated to the end of the sequence of the semantic query token corresponding to the thermal infrared image to generate the third query sequence corresponding to the thermal infrared image; Step S404: Construct the multimodal query sequence based on the first query sequence, the second query sequence, and the third query sequence.

[0044] Step S105: The semantic token redistribution module performs cross-attention reconstruction based on the multimodal query sequence and the multimodal image token to obtain a cross-modal semantic token set; Please see Figure 5 , Figure 5 yes Figure 1 The flowchart of step S105 in the example is as follows: Figure 5 As shown, in some embodiments, step S105 may include, but is not limited to, steps S501 to S504: Step S501: Perform a first cross-attention calculation based on the first query sequence corresponding to the visible light image contained in the multimodal query sequence and the first image token corresponding to the visible light image contained in the multimodal image token, and perform semantic enhancement processing on the result of the first cross-attention calculation in combination with a feedforward network to obtain the first target semantic token corresponding to the visible light image. Step S502: Perform a second cross-attention calculation based on the second query sequence corresponding to the near-infrared image contained in the multimodal query sequence and the second image token corresponding to the near-infrared image contained in the multimodal image token, and perform semantic enhancement processing on the result of the second cross-attention calculation in combination with a feedforward network to obtain the second target semantic token corresponding to the near-infrared image. Step S503: Perform third cross-attention calculation based on the third query sequence corresponding to the thermal infrared image contained in the multimodal query sequence and the third image token corresponding to the thermal infrared image contained in the multimodal image token, and perform semantic enhancement processing on the result of the third cross-attention calculation in combination with a feedforward network to obtain the third target semantic token corresponding to the thermal infrared image. Step S504: Construct the cross-modal semantic token set based on the first target semantic token, the second target semantic token, and the third target semantic token.

[0045] Specifically, the first cross-attention calculation refers to the cross-attention calculation process corresponding to the visible light image, the second cross-attention calculation refers to the cross-attention calculation process corresponding to the near-infrared image, and the third cross-attention calculation refers to the cross-attention calculation process corresponding to the thermal infrared image. For ease of distinction, the first target semantic token refers to the semantic token corresponding to the visible light image, the second target semantic token refers to the semantic token corresponding to the near-infrared image, and the third target semantic token refers to the semantic token corresponding to the thermal infrared image.

[0046] The result of cross-attention calculation is an intermediate feature representation that integrates image token information and query information.

[0047] In the semantic token redistribution module, information is extracted from the complete original token set through learnable query tokens and an adaptive redistribution mechanism, avoiding the loss of key information caused by token selection. While ensuring semantic integrity, a more compact and discriminative feature representation is generated, solving the semantic loss and feature confusion problems caused by related token selection methods.

[0048] Step S106: The cross-modal hypergraph interaction module performs cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of the multimodal image to generate global fusion features; the global fusion features are used to complete the re-identification of the target to be identified.

[0049] Please see Figure 6 , Figure 6 yes Figure 1 The flowchart of step S106 in the example is as follows: Figure 6 As shown, in some embodiments, step S106 may include, but is not limited to, steps S601 to S606: Step S601: Determine the cross-modal semantic token set as the vertex set of the hypergraph; Step S602: Calculate the semantic similarity matrix among all vertices in the vertex set, and dynamically generate a hyperedge set based on the semantic similarity matrix and a preset similarity threshold; Step S603: Construct the hypergraph based on the vertex set and the hyperedge set; Step S604: Use a hypergraph convolutional network to propagate information on the constructed hypergraph to obtain a hypergraph node information set; Step S605: Perform residual connection and feature fusion on the hypergraph node information set to obtain a fused semantic token; Step S606: Perform information aggregation processing on the fused semantic token and the global image features of the multimodal image to obtain the global fused features.

[0050] In this embodiment, a hypergraph structure is constructed, using multimodal feature tokens as vertices. Hyperedges are used to model complex high-order correlations between vertices, and information interaction is achieved through a hypergraph neural network. This overcomes the limitations of traditional pairwise interactions, enabling the capture and utilization of deeper and more complex complementary information between modalities, significantly enhancing the expressive power of features, especially demonstrating stronger performance when dealing with challenges such as occlusion and local blurring.

[0051] Steps S101 to S106 as illustrated in this embodiment involve acquiring a multimodal image of the target to be identified, wherein the multimodal image includes a visible light image, a near-infrared image, and a thermal infrared image; generating global text features based on the multimodal unified text generation strategy; generating multimodal image tokens based on the multimodal image through a segmentation-guided feature modulation module; fusing the multimodal semantic query tokens with the global text features through a semantic token redistribution module to generate a multimodal query sequence; performing cross-attention reconstruction based on the multimodal query sequence and the multimodal image tokens through the semantic token redistribution module to obtain a cross-modal semantic token set; and performing cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of the multimodal image through a cross-modal hypergraph interaction module to generate a global fusion feature; the global fusion feature is used to complete the re-identification of the target to be identified. This application's embodiments fuse image information from all modalities through a multimodal unified text generation strategy to generate a single, structured natural language description, avoiding the bias of a single modality, improving the quality of semantic information, and optimizing the computation process. A semantic token redistribution module fuses multimodal semantic query tokens with global text features and performs cross-attention reconstruction based on multimodal query sequences and multimodal image tokens, generating a more compact and discriminative feature representation while ensuring semantic integrity, thus solving the semantic loss and feature confusion problems caused by current token selection methods. A cross-modal hypergraph interaction module performs cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of multimodal images, breaking through the limitations of traditional pairwise interactions. This allows for the capture and utilization of deeper and more complex complementary information between modalities, significantly enhancing the expressive power of features, especially demonstrating stronger performance in handling challenges such as occlusion and local blurring, and significantly improving the accuracy of identity matching in complex scenarios.

[0052] To explain in detail the principles of the technical solution of this application, the overall process of this application will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principles of this application and should not be regarded as a limitation of this application.

[0053] This application provides a multimodal target re-identification method (STMI) based on segmentation guidance, unified text generation, and cross-modal hypergraph interaction, aiming to solve core problems in related technologies such as information loss, insufficient modeling of cross-modal semantic relationships, and poor quality of multimodal text generation. The method provided in this application can preserve complete semantic information, effectively model high-order cross-modal semantic relationships, and generate consistent, accurate, and computationally efficient multimodal text descriptions. The main objectives of this application include: (1) An attention modulation mechanism based on segmentation mask is proposed, which uses segmentation prior knowledge to accurately guide the model to enhance foreground features and suppress background noise, thus avoiding information loss caused by hard token selection.

[0054] (2) Design a semantic token redistribution module to extract compact and information-rich representations through learnable query tokens and adaptive redistribution mechanisms, while ensuring the semantic integrity of all original tokens.

[0055] (3) Construct a unified text generation and fusion strategy to generate a single, high-confidence structured text description by fusing all modal information, thereby solving the problems of modal inconsistency, semantic ambiguity and computational redundancy in related text generation methods.

[0056] (4) Construct a cross-modal hypergraph interaction module, regard the local features of different modalities as vertices of the hypergraph, and use hyperedges to model the complex high-order correlations between them, so as to more fully explore and utilize the complementary information between multimodalities.

[0057] By implementing the above four points, we can achieve an efficient and robust multimodal re-identification framework, which significantly improves the accuracy of identity matching in complex scenarios.

[0058] Please see Figure 7 , Figure 7 This is a system framework diagram of a multimodal target re-identification method provided in an embodiment of this application. Figure 7In this context, RGB represents visible light images, NIR represents near-infrared images, TIR represents thermal infrared images, Patch Embedding represents image embedding, Transformer Layer represents a Transformer layer, FFN (Feed-Forward Network) represents a feed-forward neural network, LN (Layer Normalization) represents layer normalization, Cross Attention represents cross-attention computation, Hyper-GCN represents a Hyper-Graph Convolutional Network, CLIP Text Encoder represents a CLIP text encoder, Unified Text represents unified text, Segmentation-Guided Feature Modulation represents segmentation-guided feature modulation, Positive represents enhanced feature locations, Negative represents suppressed feature locations, Q represents the query in the attention mechanism, K represents the key in the attention mechanism, V represents the value in the attention mechanism, Mask represents a mask, Add represents summation, and Concatenation represents concatenation. Figure 7 As shown, the STMI framework provided in this application consists of three core modules: a Segmentation-Guided Feature Modulation (SFM) module, a Semantic Token Reassignment (STR) module, and a Cross-Modal Hypergraph Interaction (CHI) module. The system receives three-modal inputs from the same target: visible light (RGB), near-infrared (NIR), and thermal infrared (TIR) ​​images. Furthermore, the system includes a preprocessing stage for multimodal unified text generation, providing high-quality semantic priors for the entire framework. All modalities share a pre-trained visual encoder (such as CLIP's ViT) and a text encoder (such as CLIP's Text Encoder) to ensure consistency in the feature space. The specific implementation details of each module and strategy are as follows: (1) Multimodal Unified Text Generation Strategy. This application first proposes a unified text generation strategy to address the problems of modal inconsistency, semantic ambiguity, and low computational efficiency in related methods. The specific implementation process is as follows: 1) Multipath Image Input: For an identity sample, the system horizontally stitches together the RGB, NIR, and TIR images of the same target to form a composite image. At the same time, the original three single-modal images are retained.

[0059] 2) Fine-grained attribute extraction: The four images (three unimodal images and one composite image) are each input into a multimodal large language model (MLLM), and specific prompt words are designed for each input path to guide the model to focus on the characteristics of different modalities (such as color, texture, and thermal radiation). The MLLM outputs a series of "attribute-value-confidence" triples for each input image.

[0060] 3) Structured text fusion: For each semantic attribute (such as "clothing color" or "whether a backpack is carried"), the system gathers all prediction results from the four input paths and selects the value with the highest confidence to fill in the data. Finally, these optimal attribute values ​​are filled into a predefined structured template to generate a single, high-confidence, and complete final text description.

[0061] 4) Text encoding: The generated uniform text (i.e. the final text description) is encoded into a global text feature vector T using CLIP’s text encoder.

[0062] The advantages of this strategy are: (a) through composite image input, the multimodal large language model (MLLM) can globally perceive multimodal information and generate more consistent descriptions; (b) through confidence fusion, it effectively utilizes the advantages of different modalities under different conditions and significantly reduces the "unknown" attribute; (c) it only needs to generate and encode one text, and the computational cost is far lower than the method of generating independent text for each modality.

[0063] (2) Segmentation-guided feature modulation (SFM) module. This module utilizes powerful segmentation prior knowledge to guide the attention mechanism of the visual encoder, achieving foreground enhancement and background suppression.

[0064] 1) Segmentation Mask Acquisition: Use a pre-trained segmentation model (such as SAM or SAM2) to generate a high-quality binary foreground segmentation mask M for each input image (i.e., visible light image, near-infrared image, and thermal infrared image).

[0065] 2) Token-level mask mapping: First, each input image is divided into N image patches, and N+1 tokens (including a category token cls) are generated for each input image. Then, based on the spatial overlap between each image patch of each input image and the corresponding binary foreground segmentation mask, a token-level binary mask vector m is generated for each input image to identify whether each token belongs to the foreground (represented by "1") or the background (represented by "0").

[0066] 3) Attention Modulation Mechanism: A learnable modulation mechanism is introduced into each layer of self-attention computation in the Transformer. Specifically: First, based on the original attention Calculate the forward modulation matrix (For enhancing foreground interaction) and negative modulation matrix (Used to suppress background interaction): ; ; in, Indicates the first The original attention of the layer, while the entire network is composed of multiple layers of attention; Secondly, based on the binary mask vector m corresponding to each input image, a foreground interaction guidance matrix R is constructed for each input image: ; Where R[i,j]=1 indicates that both tokens i and j are foreground. T represents the transpose of the binary mask vector m (where T represents the transpose). This indicates that the binary mask vector m is * A matrix of size.

[0067] Finally, modulated attention weights are calculated for each input image. : ; ; in, The attention score is represented by weights α and β, which are learnable parameters. d represents the dimension, and softmax represents the computation function. Indicates the first Attention weights after layer modulation.

[0068] 4) Robustness enhancement: During training, a "mask perturbation" mechanism is introduced to randomly flip the label of the background token to the foreground with a certain probability, so as to prevent the model from overfitting to the segmentation mask and improve the model's generalization ability.

[0069] By using the Segmentation Guided Feature Modulation (SFM) module, the model's attention is precisely guided to the target subject (such as the head and torso), significantly improving the discriminativeness and robustness of the features.

[0070] (3) Semantic Token Reassignment (STR) module. This module aims to extract compact and information-rich semantic representations to avoid information loss caused by hard token selection.

[0071] 1) Input preparation: Take the N image tokens of the input image for each modality m after processing by the SFM module. As input, m represents It has three modes: NIR and TIR.

[0072] 2) Learnable query tokens: K independent, learnable semantic query tokens are introduced into the input image for each modality m. This involves introducing semantic query tokens corresponding to visible light images (RGB), near-infrared images (NIR), and thermal infrared images (TIR).

[0073] 3) Cross-modal semantic guidance: The global text features T generated by the multimodal unified text generation strategy are appended to the semantic query token of the input image for each modality m. At the end of the sequence, an enhanced query sequence is formed. =[ For ease of distinction, the enhanced query sequence corresponding to the visible light image is called the first query sequence, the enhanced query sequence corresponding to the near-infrared image is called the second query sequence, and the enhanced query sequence corresponding to the thermal infrared image is called the third query sequence. This allows the query process to be guided by cross-modal semantic priors. This indicates that modality m corresponds to the query token sequence Q, where m represents... There are three modalities: NIR, TIR, and TIR, where T represents global text features.

[0074] 4) Cross-attention reconstruction: query sequences enhanced for each input image. As a query, each input image corresponds to an image token. Cross-attention is calculated using the input image as both key and value. After passing through a feedforward network (FFN), each input image yields a set of K+1 highly condensed semantic tokens. : ; ; in, This is an intermediate feature representation that integrates image token information and query information (i.e., the result of cross-attention calculation). , , , In this context, 'm' represents the corresponding mode, i.e. The three modes are NIR and TIR, where m∈{ , NIR, TIR}, exemplarily, It can include Corresponding semantic token semantic tokens corresponding to NIR and the semantic token corresponding to TIR .

[0075] The core advantage of the Semantic Token Redistribution (STR) module is that it actively extracts key information from all the original tokens by “redistributing” rather than “discarding” them, generating a concise representation that retains the complete semantics.

[0076] (4) Cross-modal hypergraph interaction (CHI) module. This module is responsible for modeling high-order complex semantic relationships between modalities.

[0077] 1) Hypergraph Construction: Semantic tokens generated from the three modalities processed by the STR module. , The tokens are concatenated to form a unified cross-modal semantic token set H, which serves as the vertex set V of the hypergraph.

[0078] 2) Dynamic Hyperedge Generation: Calculate the semantic similarity matrix S between all vertices. For any pair of vertices, if the similarity between each pair exceeds a preset threshold τ, then connect them to the same hyperedge e. This mechanism allows a single hyperedge to connect multiple vertices from the same or different modalities, thus naturally modeling higher-order associations (e.g., "hat" and "hair" in RGB and "head heat source" in TIR can be connected by the same hyperedge).

[0079] 3) Hypergraph Convolutional Interaction: Hypergraph convolutional interaction is responsible for modeling high-order complex semantic relationships across modalities, obtaining more complete and complex semantic information. In specific implementation, a Hypergraph Convolutional Network (Hyper-GCN) is used to propagate information on the constructed hypergraph. The information flow follows the pattern of "node-hyperedge-node". Each node aggregates information from a set of related nodes through its associated hyperedges, thus obtaining the hypergraph node information set.

[0080] 4) Residual Connections and Feature Fusion: Residual connections and feature fusion are performed on the hypergraph node information set to obtain a fused semantic token. That is, residual connections are added after each convolutional layer to preserve the original feature information. The resulting fused semantic token can capture deep-level complementary dependencies between modalities.

[0081] 5) Global Feature Enhancement: To feed high-order local semantic information back to the global representation, a cross-attention mechanism is used. Global image features G extracted from each modality are used as queries, and H is used as the key and value. Information aggregation is performed to obtain the final global fusion feature U, which is rich in high-order semantics. ; .

[0082] in, This indicates that cross-pay attention calculations are performed. For the image features corresponding to the modality, This indicates that the matrix size is 3*D.

[0083] Finally, regarding the model training and optimization in this embodiment, the specific steps are as follows: The model training employs multi-task supervised learning. Based on the key feature representations: global image features G, fused semantic features U, and global text features T, their label smoothing cross-entropy loss and triplet loss are calculated respectively. The final total loss function is the sum of these three, and the specific calculation formula is as follows: ; ; in, Indicates the characteristics of the corresponding mode. This represents the loss for the corresponding mode. This represents the label smoothing cross-entropy loss. This represents triplet loss. The loss represents the global image features G. This represents the loss of the fused semantic features U. Let T represent the loss of global text features, and L represent the total loss.

[0084] This design ensures that feature representations at different levels can be fully optimized, thereby improving the model's re-identification performance.

[0085] It should be noted that this embodiment is only a brief illustrative description of the overall process of a multimodal target re-identification method. The detailed description of each step can be referred to the relevant content in the foregoing embodiments, and will not be repeated here. It is understood that this application does not impose any limitations on this.

[0086] This application embodiment acquires a multimodal image of the target to be identified; wherein the multimodal image includes a visible light image, a near-infrared image, and a thermal infrared image; based on a multimodal unified text generation strategy, global text features are generated from the multimodal image; a segmentation-guided feature modulation module generates a multimodal image token from the multimodal image; a semantic token redistribution module fuses the multimodal semantic query token with the global text features to generate a multimodal query sequence; the semantic token redistribution module performs cross-attention reconstruction based on the multimodal query sequence and the multimodal image token to obtain a cross-modal semantic token set; and a cross-modal hypergraph interaction module performs cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of the multimodal image to generate a global fusion feature; the global fusion feature is used to complete the re-identification of the target to be identified. This application's embodiments fuse image information from all modalities through a multimodal unified text generation strategy to generate a single, structured natural language description, avoiding the bias of a single modality, improving the quality of semantic information, and optimizing the computation process. A semantic token redistribution module fuses multimodal semantic query tokens with global text features and performs cross-attention reconstruction based on multimodal query sequences and multimodal image tokens, generating a more compact and discriminative feature representation while ensuring semantic integrity, thus solving the semantic loss and feature confusion problems caused by current token selection methods. A cross-modal hypergraph interaction module performs cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of multimodal images, breaking through the limitations of traditional pairwise interactions. This allows for the capture and utilization of deeper and more complex complementary information between modalities, significantly enhancing the expressive power of features, especially demonstrating stronger performance in handling challenges such as occlusion and local blurring, and significantly improving the accuracy of identity matching in complex scenarios.

[0087] To comprehensively evaluate the effectiveness of the STMI method proposed in the embodiments of this application, performance verification was performed on publicly available multimodal object re-identification datasets (RGBNT201, RGBNT100, and MSVR310). The standard evaluation metrics used in the experiment were the mean accuracy (mAP) and cumulative matching curve (CMC) at Rank-1, 5, and 10.

[0088] (1) Comparative experiment with related technologies: 1) Performance on the pedestrian re-identification dataset RGBNT201: As shown in Table 1, the STMI method proposed in this application achieves an mAP of 81.2%, which is the best performance among all compared methods. Specifically, the STMI method proposed in this application surpasses the previous best method IDEA by 1.0% in mAP. Compared with TOP-ReID (72.3%) and EDITOR (66.5%), STMI achieves significant improvements of 8.9% and 14.7% in mAP, respectively. These results demonstrate the effectiveness of the STMI method proposed in this application in enhancing cross-modal semantic alignment and maintaining the integrity of tag-level representations.

[0089] Table 1: STMI's performance on RGBNT201

[0090] 2) Performance on the vehicle re-identification datasets RGBNT100 and MSVR310: As shown in Table 2, the STMI method proposed in this application achieves an mAP of 89.1% on the RGBNT100 dataset, surpassing strong baselines such as IDEA (87.2%) and DeMo (86.2%). On the more challenging MSVR310 dataset, the STMI method proposed in this application achieves an mAP of 64.8%, which is 17.8% higher than the previous best result of IDEA (47.0%). These results highlight the effectiveness and robustness of the STMI method proposed in this application under complex conditions such as cluttered backgrounds, occlusion, and modal inconsistencies.

[0091] Table 2: STMI's performance on RGBNT100 and MSVR310

[0092] (2) Ablation experiments of each module: To verify the independent contribution of each core module in this application, a systematic ablation study was conducted on the RGBNT201 and MSVR310 datasets in the embodiments of this application. The results are shown in Tables 3 and 4 below: Table 3: Ablation Experiment Results of STMI on RGBNT201

[0093] Table 4: Ablation Experiment Results of STMI on MSVR310

[0094] Table 3 shows the performance changes on the RGBNT201 dataset, starting with the baseline model (Model A, mAP 70.3%), as core modules were gradually added. First, after introducing the Segmentation Guided Feature Modulation (SFM) module (Model B), the mAP significantly improved to 76.1%. This improvement strongly demonstrates that using high-quality segmentation prior knowledge to guide the attention mechanism can effectively enhance the feature representation of the foreground target and suppress the interference of background noise, thus significantly improving the overall performance of the model. Building on this, the Semantic Token Reassignment (STR) module (Model C) was further added, and the mAP continued to improve to 78.1%. This result shows that the STR module adaptively reassigns complete token information through learnable query tokens, successfully avoiding information loss caused by hard token selection, and extracting more compact and discriminative semantic representations while preserving fine-grained visual details. Finally, integrating all modules (Model D), the mAP reached 81.2%. This series of progressive performance gains clearly demonstrates that the Segmentation Guided Feature Modulation (SFM) module, Semantic Token Reassignment (STR) module, and Cross-Modal Hypergraph Interaction (CHI) module are reasonably designed and have significant synergistic effects, together constituting the excellent performance of the STMI method proposed in this application embodiment.

[0095] Furthermore, the ablation experiment results on the MSVR310 dataset in Table 4 further demonstrate the universality and effectiveness of the embodiments of this application. On this more challenging vehicle re-identification dataset, the embodiments of this application also exhibit strong performance improvement capabilities. The experimental results verify that each module can function stably under different datasets and task scenarios, fully demonstrating that the STMI method proposed in the embodiments of this application has good generalization ability and wide applicability.

[0096] To explore the interpretability and robustness of this application, embodiments of this application visualize attention maps on the RGBNT201 and MSVR310 datasets. See also... Figures 8 to 9 , Figure 8 This application provides a multimodal target re-identification method with a vehicle visualization heatmap on a dataset. Figure 8 The dataset used in this paper is the RGBNT201 dataset. Figure 9 is a pedestrian visualization heatmap on the dataset for a multimodal target re-identification method provided in this embodiment of the application. Figure 9 The dataset used in this study is the MSVR310 dataset, such as... Figure 8 and Figure 9As shown, the STMI method provided in this application consistently focuses on identity-related regions, such as the upper body, legs, and carried items in pedestrian re-identification, and vehicle license plates, wheels, and outlines in vehicle re-identification. It also suppresses specific modal noise, such as infrared blurring and thermal saturation. Figure 8 and Figure 9 As can be seen, the STMI method provided in this application generates a clearer and more focused attention map, verifying its effective cross-modal feature modeling capability.

[0097] Therefore, the multimodal target re-identification method provided in this application has the following key advantages: (1) Technical point one, segmented guided attention modulation (SFM): Key technical points: The mask generated by the pre-trained segmentation model is used as prior knowledge to guide the attention mechanism to weight tokens in multimodal images, thereby enhancing the foreground region and suppressing background noise.

[0098] Beneficial effects: Compared with methods that directly use segmentation maps as auxiliary input or perform hard pruning, the embodiments of this application deeply integrate segmentation information into attention calculation, achieving more refined and consistent feature enhancement, and effectively improving the robustness of the model in complex backgrounds.

[0099] (2) Technical point two, semantic token reallocation (STR): Key technical points: By using learnable query tokens and an adaptive redistribution mechanism, information is extracted from the complete original token set, avoiding the loss of critical information due to token selection.

[0100] Beneficial effects: While ensuring semantic integrity, it generates more compact and discriminative feature representations, solving the semantic loss and feature confusion problems caused by related token selection methods.

[0101] (3) Technical point three: Unifying the fusion of text generation and confidence perception: Key technical points: Integrating image information from all modalities to generate a single, structured natural language description. A confidence-aware imputation strategy is used to supplement missing information from low-confidence modalities by leveraging attributes of high-confidence modalities.

[0102] Beneficial effects: It fundamentally solves the three major pain points of related text generation methods: 1) Modal consistency: The description integrates complementary information from all modalities, avoiding the bias of a single modality; 2) Semantic clarity: Through confidence filling, it significantly reduces uncertain words such as "unknown" and "fuzzy", generating more reliable and information-rich descriptions; 3) Computational efficiency: It generates and encodes only one text, avoiding the computational redundancy caused by multiple text descriptions, and significantly improving training and inference efficiency.

[0103] (4) Technical Point Four: Cross-Modal Hypergraph Interaction (CHI): Key technical points: Construct a hypergraph structure, use multimodal feature tokens as vertices, utilize hyperedges to model complex high-order correlations between vertices, and use a hypergraph neural network for information interaction.

[0104] Beneficial effects: It breaks through the limitations of traditional pairwise interactions, and can capture and utilize deeper and more complex complementary information between modalities, significantly enhancing the expressive power of features, especially showing stronger performance when dealing with challenges such as occlusion and local blur.

[0105] In summary, this application provides a novel and efficient multimodal re-identification solution through the synergistic effect of four core modules: segmentation guidance, token redistribution, unified text generation, and hypergraph interaction. In particular, the unified text generation strategy not only improves the quality of semantic information but also optimizes the computational process, enabling this application to achieve superior performance compared to related technologies in multiple publicly available benchmark tests. This demonstrates significant theoretical value and broad application prospects.

[0106] Please see Figure 10 This application also provides a multimodal target re-identification device 1000, which can implement the above-described method. The device includes the following modules: The multimodal image acquisition module 1001 is used to acquire multimodal images of the target to be identified; wherein, the multimodal images include visible light images, near-infrared images, and thermal infrared images; The global text feature generation module 1002 is used to generate global text features based on the multimodal image according to the multimodal unified text generation strategy. The multimodal image token generation module 1003 is used to generate a multimodal image token based on the multimodal image through the segmentation-guided feature modulation module; The multimodal query sequence generation module 1004 is used to fuse the multimodal semantic query token with the global text features through the semantic token redistribution module to generate a multimodal query sequence; The cross-attention reconstruction module 1005 is used to perform cross-attention reconstruction based on the multimodal query sequence and the multimodal image token through the semantic token redistribution module to obtain a cross-modal semantic token set; The hypergraph interaction operation module 1006 is used to perform cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of the multimodal image through the cross-modal hypergraph interaction module, and generate global fusion features; the global fusion features are used to complete the re-identification of the target to be identified.

[0107] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0108] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0109] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0110] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1102 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and is called and executed by the processor 1101 using the methods described in the embodiments of this application. Input / output interface 1103 is used to implement information input and output; The communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104); The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.

[0111] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0112] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0113] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0114] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0116] This application provides a multimodal target re-identification method, apparatus, electronic device, storage medium, and program product, which acquires multimodal images of the target to be identified; wherein the multimodal images include visible light images, near-infrared images, and thermal infrared images; generates global text features based on the multimodal unified text generation strategy; generates multimodal image tokens based on the multimodal images through a segmentation-guided feature modulation module; fuses the multimodal semantic query tokens with the global text features through a semantic token redistribution module to generate a multimodal query sequence; performs cross-attention reconstruction based on the multimodal query sequence and the multimodal image tokens through the semantic token redistribution module to obtain a cross-modal semantic token set; and performs cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of the multimodal images through a cross-modal hypergraph interaction module to generate global fusion features; the global fusion features are used to complete the re-identification of the target to be identified. This application's embodiments fuse image information from all modalities through a multimodal unified text generation strategy to generate a single, structured natural language description, avoiding the bias of a single modality, improving the quality of semantic information, and optimizing the computation process. A semantic token redistribution module fuses multimodal semantic query tokens with global text features and performs cross-attention reconstruction based on multimodal query sequences and multimodal image tokens, generating a more compact and discriminative feature representation while ensuring semantic integrity, thus solving the semantic loss and feature confusion problems caused by current token selection methods. A cross-modal hypergraph interaction module performs cross-modal hypergraph interaction operations based on the cross-modal semantic token set and the global image features of multimodal images, breaking through the limitations of traditional pairwise interactions. This allows for the capture and utilization of deeper and more complex complementary information between modalities, significantly enhancing the expressive power of features, especially demonstrating stronger performance in handling challenges such as occlusion and local blurring, and significantly improving the accuracy of identity matching in complex scenarios.

[0117] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A multi-modal target re-identification method, characterized in that, The method comprises the following steps: Obtain a multi-modal image of a target to be identified; wherein the multi-modal image comprises a visible light image, a near-infrared image and a thermal infrared image; Based on a multi-modal unified text generation strategy, generate a global text feature according to the multi-modal image; Through a segmentation guided feature modulation module, generate a multi-modal image token according to the multi-modal image; Through a semantic token reassignment module, fuse a multi-modal semantic query token with the global text feature to generate a multi-modal query sequence; Through the semantic token reassignment module, cross attention reconstruction is performed according to the multi-modal query sequence and the multi-modal image token to obtain a cross-modal semantic token set; Through a cross-modal hypergraph interaction module, cross-modal hypergraph interaction is performed according to the cross-modal semantic token set and the global image feature of the multi-modal image to generate a global fusion feature; the global fusion feature is used to complete the re-identification of the target to be identified.

2. The method of claim 1, wherein, Based on the multi-modal unified text generation strategy, the global text feature is generated according to the multi-modal image, which comprises: The visible light image, the near-infrared image and the thermal infrared image in the multi-modal image are horizontally spliced to generate a composite image; The visible light image, the near-infrared image, the thermal infrared image and the composite image are respectively input into a multi-modal large language model to output the semantic attribute prediction results corresponding to the visible light image, the near-infrared image, the thermal infrared image and the composite image; wherein each image input path is designed with a target prompt word to guide the multi-modal large language model to focus on the characteristics of different modalities according to the target prompt word; From the semantic attribute prediction results corresponding to the visible light image, the near-infrared image, the thermal infrared image and the composite image, the corresponding target attribute values are selected; The target attribute values corresponding to the visible light image, the near-infrared image, the thermal infrared image and the composite image are input into a pre-defined structured template to generate global text description information; The global text description information is encoded by a text encoder to obtain the global text feature.

3. The method of claim 1, wherein, The method further comprises: Through a pre-training segmentation model in the segmentation guided feature modulation module, mask segmentation is performed on the visible light image, the near-infrared image and the thermal infrared image respectively to obtain a set of binary foreground segmentation masks; Image segmentation is performed on the visible light image, the near-infrared image and the thermal infrared image respectively to obtain image blocks corresponding to the visible light image, the near-infrared image and the thermal infrared image; According to the image blocks of the visible light image and the binary foreground segmentation mask of the visible light image, a binary mask vector of the visible light image is generated; According to the image blocks of the near-infrared image and the binary foreground segmentation mask of the near-infrared image, a binary mask vector of the near-infrared image is generated; According to the image blocks of the thermal infrared image and the binary foreground segmentation mask of the thermal infrared image, a binary mask vector of the thermal infrared image is generated; The original attention of the attention modulation mechanism in the segmentation guidance feature modulation module is used to calculate a positive modulation matrix and a negative modulation matrix; the positive modulation matrix is used to enhance foreground interaction, and the negative modulation matrix is used to suppress background interaction; A corresponding foreground interaction guidance matrix is constructed according to a binary mask vector corresponding to the visible light image, the near-infrared image and the thermal infrared image; According to the positive modulation matrix and the negative modulation matrix, and in combination with the foreground interaction guidance matrix corresponding to the visible light image, the near-infrared image and the thermal infrared image, the attention weight corresponding to the visible light image, the near-infrared image and the thermal infrared image is calculated; the attention weight is used to guide the attention of the model to the target subject.

4. The method of claim 1, wherein, The multi-modal semantic query token includes a semantic query token corresponding to the visible light image, a semantic query token corresponding to the near-infrared image and a semantic query token corresponding to the thermal infrared image, and the multi-modal semantic query token is fused with the global text feature by the semantic token redistribution module to generate a multi-modal query sequence, including: The global text feature is spliced to the sequence tail of the semantic query token corresponding to the visible light image to generate a first query sequence corresponding to the visible light image; The global text feature is spliced to the sequence tail of the semantic query token corresponding to the near-infrared image to generate a second query sequence corresponding to the near-infrared image; The global text feature is spliced to the sequence tail of the semantic query token corresponding to the thermal infrared image to generate a third query sequence corresponding to the thermal infrared image; The multi-modal query sequence is constructed according to the first query sequence, the second query sequence and the third query sequence.

5. The method of claim 1, wherein, According to the multi-modal query sequence and the multi-modal image token, cross-attention reconstruction is performed by the semantic token redistribution module to obtain a cross-modal semantic token set, including: First cross-attention calculation is performed on the first query sequence corresponding to the visible light image included in the multi-modal query sequence and the first image token corresponding to the visible light image included in the multi-modal image token, and the result of the first cross-attention calculation is processed by a feedforward network for semantic enhancement to obtain a first target semantic token corresponding to the visible light image; Second cross-attention calculation is performed on the second query sequence corresponding to the near-infrared image included in the multi-modal query sequence and the second image token corresponding to the near-infrared image included in the multi-modal image token, and the result of the second cross-attention calculation is processed by a feedforward network for semantic enhancement to obtain a second target semantic token corresponding to the near-infrared image; perform third cross-attention calculation on the third query sequence corresponding to the thermal infrared image included in the multi-modal query sequence and the third image token corresponding to the thermal infrared image included in the multi-modal image token, and perform semantic enhancement processing on the result of the third cross-attention calculation in combination with a feedforward network to obtain a third target semantic token corresponding to the thermal infrared image; construct the cross-modal semantic token set according to the first target semantic token, the second target semantic token and the third target semantic token.

6. The method of claim 1, wherein, The cross-modal hypergraph interaction module performs cross-modal hypergraph interaction operation on the cross-modal semantic token set and the global image feature of the multi-modal image to generate a global fusion feature, including: determining the cross-modal semantic token set as a vertex set of a hypergraph; calculating a semantic similarity matrix between all vertices in the vertex set, and dynamically generating a hyperedge set according to the semantic similarity matrix and a preset similarity threshold; constructing the hypergraph according to the vertex set and the hyperedge set; performing information propagation on the constructed hypergraph by using a hypergraph convolution network to obtain a hypergraph node information set; performing residual connection and feature fusion on the hypergraph node information set to obtain a fusion semantic token; performing information aggregation processing on the fusion semantic token and the global image feature of the multi-modal image to obtain the global fusion feature.

7. A multi-modal target re-identification device, comprising: The device includes the following modules: A multi-modal image acquisition module is configured to acquire a multi-modal image of a target to be recognized, wherein the multi-modal image includes a visible light image, a near-infrared image and a thermal infrared image. A global text feature generation module is configured to generate a global text feature based on a multi-modal unified text generation strategy according to the multi-modal image. A multi-modal image token generation module is configured to generate a multi-modal image token from the multi-modal image by using a segmentation-guided feature modulation module. A multi-modal query sequence generation module is configured to fuse a multi-modal semantic query token with the global text feature by using a semantic token reassignment module to generate a multi-modal query sequence. A cross-attention reconstruction module is configured to perform cross-attention reconstruction on the multi-modal query sequence and the multi-modal image token by using the semantic token reassignment module to obtain a cross-modal semantic token set. A hypergraph interaction operation module is configured to perform cross-modal hypergraph interaction operation on the cross-modal semantic token set and the global image feature of the multi-modal image by using a cross-modal hypergraph interaction module to generate a global fusion feature, wherein the global fusion feature is used to complete the re-identification of the target to be recognized.

8. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method of any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the method of any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-6.

Images