Multi-modal target identification method and system based on comprehensive prompt and Mama deep interaction
By adopting a multimodal target recognition method based on comprehensive prompts and Mamba deep interaction, the problem of performance degradation of traditional multimodal visual recognition systems in complex environments is solved, and efficient recognition and robustness improvement of multimodal targets are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-02
- Publication Date
- 2026-03-13
AI Technical Summary
Traditional single-modal visual recognition systems exhibit significantly reduced recognition performance in nighttime, bright light, occlusion, or complex weather environments. Multimodal feature fusion methods struggle to balance modal saliency information with global semantic diversity, thus limiting the performance improvement of cross-modal visual recognition models.
A multimodal target recognition method based on comprehensive prompts and deep Mamba interaction is adopted. Through preprocessing, deep learning network model, salient-random prompt aggregation module, Transformer encoding module and dual-path adaptive Mamba module, deep unified representation and feature fusion of multimodal image data are achieved, combined with joint constraint loss optimization model.
It improves the consistency and robustness of cross-modal representations, significantly enhances the re-identification accuracy and robustness of multimodal targets, and enables stable multimodal target recognition in complex environments.
Smart Images
Figure CN121661449A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a multimodal target recognition method and system based on comprehensive cues and Mamba deep interaction. Background Technology
[0002] With the rapid development of intelligent security, autonomous driving, and unmanned systems, multimodal perception has gradually become an important research direction in the field of visual recognition. Traditional single-modal recognition systems mainly rely on visible light images, achieving high recognition accuracy under sufficient daylight conditions. However, in nighttime, strong light, occlusion, or complex weather environments, the visible information of the image is severely limited, leading to a significant decline in recognition performance. To overcome this problem, researchers have introduced near-infrared and thermal infrared modes, achieving stable target recognition through complementary information in different spectral ranges, thereby significantly improving the system's robustness across illumination and environmental adaptability. With the development of deep learning, convolutional neural networks and Transformers have achieved significant breakthroughs in single-modal vision tasks, with recognition accuracy approaching or even exceeding human levels. However, in multimodal tasks, due to significant differences in modality distribution, direct feature fusion often leads to information redundancy and conflicts, limiting the improvement of model performance. To alleviate this problem, researchers have proposed various feature fusion strategies, such as unsupervised feature learning based on clustering by Li et al., consistency modeling based on contrastive learning by Chen et al., and pseudo-label optimization using student-teacher networks by Zhang et al. These methods have made some progress in feature alignment and semantic fusion, but in scenarios with large spectral differences, the models still struggle to capture key discriminative features and are quite sensitive to modal differences. Recently, cue learning and state-space models have provided new ideas for cross-modal feature alignment. Cue learning enhances the model's task adaptability through explicit semantic guidance, while state-space models improve global dependency modeling performance with their efficient long-sequence modeling capabilities. However, traditional cue learning still struggles to balance modal saliency information and global semantic diversity, and existing multimodal fusion methods are insufficient in distinguishing between modality-specific and shared information, leading to feature confusion and unstable semantic representation, which has become a key problem that needs to be addressed in cross-modal visual recognition. Summary of the Invention
[0003] To address the technical problems existing in the background art, the present invention provides a multimodal target recognition method based on comprehensive prompts and Mamba deep interaction.
[0004] In a first aspect, the present invention provides a multimodal target recognition method based on comprehensive prompts and Mamba deep interaction, comprising the following steps: Step 1: Preprocess the multimodal image data; the multimodal image data includes: visible light image data, near-infrared image data, and thermal infrared image data; Step 2: Process the preprocessed multimodal image data using a deep learning network model to obtain a deep unified representation of the multimodal image data; Step 3: Based on the deep unified representation of multimodal image data, perform target matching and recognition; Step two includes: inputting the preprocessed multimodal image data into a pre-trained CLIP visual coding model for feature embedding to obtain the basic modal feature representation of each modal image data; Based on the basic modal feature representations of each modal image data, an enhanced modal feature representation of each modal image data is generated through a salient-random cue aggregation module; The enhanced modal feature representations of each modality image data are input into the Transformer encoding module to obtain a unified high-dimensional semantic representation across modalities; By using a dual-path adaptive Mamba module, the high-dimensional semantic representations of image data from various modalities are decoupled and fused, and modality-specific and modality-shared features of each modality are extracted to achieve deep cross-modal interaction and feature fusion, thereby obtaining a deep unified representation of multimodal image data.
[0005] Furthermore, the method also includes: Joint constraints are used to optimize deep learning network models.
[0006] Furthermore, the step of inputting the preprocessed multimodal image data into a pre-trained CLIP visual coding model for feature embedding to obtain the basic modal feature representation of each modal image data includes: The pre-trained CLIP visual coding model performs region segmentation and feature extraction on the preprocessed input image data of each modality to obtain patch-level feature sequences that represent the local semantic information of each modality image data; Linear mapping and embedding are performed on the patch-level feature sequences representing local semantic information of each modality image data to form a basic modal feature representation of each modality image data with a unified dimension.
[0007] Furthermore, the basic modal feature representation based on each modal image data, through the saliency-random cue aggregation module, generates enhanced modal feature representations for each modal image data, including: Based on the L2 norm of the basic modal feature representation of each modal image data, salient features are selected; Random prompts are generated by randomly selecting several basic modal feature representations from the basic modal feature representations of each modal image data. By concatenating random cues with prominent cues, a comprehensive cues with modality invariance are generated; The generated comprehensive prompts are concatenated with the basic modal feature representations of each modal image data to obtain the enhanced modal feature representations of each modal image data.
[0008] Furthermore, the step of inputting the enhanced modal feature representations of each modality image data into the Transformer encoding module to obtain a cross-modal unified high-dimensional semantic representation includes: Linear projection and normalization were performed on the image data for each modality; A multi-head self-attention mechanism is performed on the normalized image data of each modality to achieve feature interaction and semantic alignment within and between the image data of each modality; The image data of each modality after the multi-head self-attention mechanism has been executed is fed into the feedforward network and combined with residual connections to obtain the high-dimensional semantic representation of the image data of each modality.
[0009] Furthermore, the dual-path adaptive Mamba module includes modality-specific paths and modality-shared paths; The modality-specific path includes multiple first Mamba sub-modules, which are used to preserve local discriminative features using depthwise convolution; The modal sharing path includes multiple second Mamba submodules; these second Mamba submodules obtain global consistency information through state space scanning.
[0010] Furthermore, the step of decoupling and fusing the high-dimensional semantic representations of each modality's image data using a dual-path adaptive Mamba module, extracting modality-specific and modality-shared features from each modality's image data, and achieving deep cross-modal interaction and feature fusion to obtain a deep unified representation of multimodal image data includes: The modality-specific path models the high-dimensional semantic representation of each modality image data through the first Mamba submodule and extracts the modality-specific features of each modality image data; The modality sharing path models the high-dimensional semantic representation of each modality image data through the second Mamba submodule and extracts the modality sharing features between each modality image data. Dynamic weighted fusion is performed on modality-shared features and modality-specific features to obtain global multimodal features; Linear projection and normalization are performed on global multimodal features to obtain a deep unified representation of multimodal image data.
[0011] Furthermore, step three includes: Define a label-smooth cross-entropy loss to achieve classification constraints; Define triplet loss to achieve metric constraints; A global joint optimization objective function is constructed to achieve end-to-end training.
[0012] Secondly, the present invention provides a multimodal target recognition system based on comprehensive prompts and Mamba deep interaction, comprising: The preprocessing module is used to preprocess multimodal image data; Deep learning network models are used to process preprocessed multimodal image data to obtain a deep unified representation of the multimodal image data. The target matching and recognition module is used for target matching and recognition based on the deep unified representation of multimodal image data; The deep learning network model includes: a pre-trained CLIP visual coding model, used to embed features into preprocessed multimodal image data to obtain basic modal feature representations of each modal image data; The salient-random suggestion aggregation module is used to generate enhanced modal feature representations for each modal image data based on the basic modal feature representations of each modal image data. The Transformer encoding module is used to enhance modal feature representation based on image data of each modality and obtain a unified high-dimensional semantic representation across modalities. The dual-path adaptive Mamba module is used to decouple and fuse features of high-dimensional semantic representations of image data from various modalities, extract modality-specific features and modality-shared features of each modality of image data, realize deep cross-modal interaction and feature fusion, and obtain a deep unified representation of multimodal image data.
[0013] Thirdly, the present invention provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the computer-readable storage medium is located to execute the multimodal target recognition method based on comprehensive prompts and Mamba deep interaction described above.
[0014] The beneficial effects of this invention are as follows: (1) This invention adaptively extracts salient and random cues from different modalities in the salient-random cue aggregation module, fully exploring target features from the perspectives of saliency and global diversity, enabling the deep learning network model to focus on more discriminative cross-modal information. At the same time, the dual-path adaptive Mamba module enables parallel learning of modality-specific features and modality-shared features, effectively solving the problem of intra-class feature differences caused by different sensor imaging conditions, allowing the model to adaptively balance the feature contributions of different modalities in the feature space, and improving the consistency and robustness of cross-modal representation.
[0015] (2) The present invention adopts a dual constraint mechanism of joint label smooth cross-entropy loss and triplet loss to jointly optimize the deep learning network model from both classification and measurement aspects. This not only enhances the inter-class discriminability but also reduces the intra-class gap, thereby significantly improving the re-identification accuracy and robustness of multimodal targets. Attached Figure Description
[0016] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0017] Figure 1 This is a flowchart of a multimodal target recognition method based on Mamba interaction and integrated prompting learning according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structural framework of a multimodal target recognition model based on Mamba interaction and integrated prompting learning according to an embodiment of the present invention; Figure 3 A flowchart illustrating how an enhanced modal feature representation of each modality image data is generated using a salient-random cue aggregation module according to an embodiment of the present invention; Figure 4 A flowchart for obtaining a cross-modal unified high-dimensional semantic representation according to an embodiment of the present invention; Figure 5 This is a flowchart illustrating the process of obtaining a deep unified representation of multimodal image data according to an embodiment of the present invention. Detailed Implementation
[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0019] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, each technical and scientific term used in these embodiments has the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0020] Example 1: like Figure 1 and Figure 2 As shown, this embodiment provides a multimodal target recognition method based on comprehensive prompts and Mamba deep interaction, including: S1: Preprocess the multimodal image data and divide the preprocessed multimodal image data into a training set and a validation set; the multimodal image data includes: visible light image data, near-infrared image data and thermal infrared image data; Specifically, data augmentation (e.g., random horizontal flipping, cropping, and erasing) and normalization are performed on each modality of the multimodal image data.
[0021] S2: Train the deep learning network model using the training set, optimize the parameters of the deep learning network model, and enable it to output a deep unified representation with strong discriminative power for the input multimodal image data.
[0022] The deep learning network models include: a pre-trained CLIP visual encoding model, a salient-random cue aggregation module, a Transformer encoding module, and a dual-path adaptive Mamba module; Specifically, the following steps are included: S21: Input the preprocessed multimodal image data into the pre-trained CLIP visual coding model for feature embedding to obtain the basic modal feature representation of each modal image data; Specifically, the following steps are included: S21-1: The pre-trained CLIP visual coding model performs region segmentation and feature extraction on the pre-processed input modal image data to obtain patch-level feature sequences that represent the local semantic information of each modal image data; Specifically, the pre-trained CLIP visual coding model performs region segmentation and feature extraction on the input preprocessed visible light image data, near-infrared image data, and thermal infrared image data to obtain patch-level feature sequences representing local semantic information of the visible light image data, near-infrared image data, and thermal infrared image data.
[0023] S21-2: Perform linear mapping and embedding on the patch-level feature sequences representing local semantic information of each modality image data to form a unified-dimensional basic modal feature representation of each modality image data; Specifically, the patch-level feature sequences representing local semantic information of visible light image data, near-infrared image data, and thermal infrared image data are linearly mapped and embedded to form a unified dimension of visible light basic modal feature representation, near-infrared basic modal feature representation, and thermal infrared basic modal feature representation; Feature embedding through the CLIP visual coding model ensures that image data from each modality has a consistent dimensional representation in the feature space, providing structured input for subsequent cross-modal feature alignment and fusion.
[0024] S22: Based on the basic modal feature representation of each modal image data, an enhanced modal feature representation of each modal image data is generated through a salient-random cue aggregation module; Specifically, such as Figure 3 As shown, it includes the following steps: S22-1: L2 norm based on the basic modal feature representation of each modal image data, and selection of salient features; A: Calculate the L2 norm of the visible light fundamental mode feature representation, the near-infrared fundamental mode feature representation, and the thermal infrared fundamental mode feature representation respectively, in order to measure the energy response intensity of the visible light fundamental mode feature representation, the near-infrared fundamental mode feature representation, and the thermal infrared fundamental mode feature representation; A2: Based on the L2 norm of the basic modal feature representations of each modal image data, the basic modal feature representations of the top k energy response intensities are selected as salient cues to highlight the discriminative feature regions and discriminative semantic information of the target subject in different modalities; Specifically, when extracting the cue information from the visible light fundamental mode feature representation, based on the L2 norm of the near-infrared fundamental mode feature representation and the thermal infrared fundamental mode feature representation, the top k energy response intensities of the fundamental mode feature representation from each of the near-infrared fundamental mode feature representation and the thermal infrared fundamental mode feature representation are selected as significant cue information, thereby enhancing the discriminative power of the visible light mode features.
[0025] S22-2: Randomly extract several basic modal feature representations from the basic modal feature representations of each modal image data to form random prompts; Specifically, random cueing is constructed by randomly selecting several basic modal feature representations from the visible light, near-infrared, and thermal infrared basic modal feature representations, respectively. By randomly selecting several basic modal feature representations from the basic modal feature representations of each modal image data as random cueing, global information is supplemented and the generalization ability of the model is improved.
[0026] S22-3: Combine random cues with prominent cues to generate comprehensive cues with modality invariance, thereby enhancing the integrity of cross-modal information.
[0027] By concatenating salient and random cues from different modalities to form a comprehensive cue, the differences and diversity between modalities are integrated in terms of both saliency and randomness. This not only achieves dynamic cue enhancement for multimodal inputs, enabling the comprehensive cue to simultaneously retain key discrimination regions and overall semantic background information, but also avoids the model's over-reliance on a few highly activated regions and improves global diversity, providing cross-modal contextual correlation signals for subsequent Transformer encoding.
[0028] S22-4: The generated comprehensive prompts are concatenated with the basic modal feature representations of each modal image data to obtain the enhanced modal feature representations of each modal image data.
[0029] Specifically, the generated integrated prompts are concatenated with the visible light basic mode feature representation, the near-infrared basic mode feature representation, and the thermal infrared basic mode feature representation to obtain the visible light enhanced mode feature representation, the near-infrared enhanced mode feature representation, and the thermal infrared mode feature; Among them, the obtained visible light enhanced mode feature representation, near-infrared enhanced mode feature representation, and thermal infrared mode feature all include basic mode feature representation and comprehensive hints; In this embodiment, the dimension of the enhanced modal feature representation of each modal image data is . That is, the dimensions of the visible light enhanced mode feature representation, the near-infrared enhanced mode feature representation, and the thermal infrared mode feature are: ,in This is the sum of the number of prominent prompts and random prompts.
[0030] This enhanced modal feature representation, as input to the subsequent Transformer encoding module, can establish deep interaction relationships between different modalities and improve the feature alignment effect between modalities.
[0031] S23: Input the enhanced modal feature representations of each modality image data into the Transformer encoding module to obtain a unified high-dimensional semantic representation across modalities; Specifically, the Transformer encoding module performs deep representation learning on the received visible light enhanced modal feature representation, near-infrared enhanced modal feature representation, and thermal infrared modal feature, capturing cross-modal semantic associations and global dependencies, thereby obtaining a unified high-dimensional semantic representation across modalities.
[0032] Specifically, such as Figure 4 As shown, it includes the following steps: S23-1: Perform linear projection and normalization processing on the image data of each modality; Specifically, to maintain the consistency of the visible light enhanced modal feature representation, near-infrared enhanced modal feature representation, and thermal infrared modal feature on the numerical scale, a linear mapping is first performed on the visible light enhanced modal feature representation, near-infrared enhanced modal feature representation, and thermal infrared modal feature, and the dimensions of the visible light enhanced modal feature representation, near-infrared enhanced modal feature representation, and thermal infrared enhanced modal feature representation are standardized to a unified embedding dimension (e.g., dimension is 1). Subsequently, the visible light enhanced modal feature representation, near-infrared enhanced modal feature representation, and thermal infrared modal feature, which are standardized to a unified embedding dimension, are normalized to stabilize the feature distribution and improve training convergence. S23-2: Perform a multi-head self-attention mechanism on the normalized image data of each modality to achieve feature interaction and semantic alignment within and between the image data of each modality; Specifically, a multi-head self-attention mechanism is used to perform weighted aggregation of the normalized visible light enhanced modal feature representation, near-infrared enhanced modal feature representation, and thermal infrared enhanced modal feature representation.
[0033] Each attention head can independently learn different semantic relevance weights, and multiple attention heads are computed in parallel to simultaneously focus on the local features and global contextual information of each modal image data (visible light enhanced modal feature representation, near-infrared enhanced modal feature representation, and thermal infrared enhanced modal feature representation). This enables the extraction of fine-grained information within the visible light enhanced modal feature representation, near-infrared enhanced modal feature representation, and thermal infrared enhanced modal feature representation, as well as the semantic alignment among them.
[0034] S23-3: The image data of each modality after the multi-head self-attention mechanism has been executed is fed into the feedforward network and combined with residual connections to obtain the high-dimensional semantic representation of the image data of each modality; Among them, the high-dimensional semantic representations of each modality of image data are visible light high-dimensional semantic representation, near-infrared high-dimensional semantic representation and thermal infrared high-dimensional semantic representation, respectively; Specifically, the visible light enhanced modal feature representation, near-infrared enhanced modal feature representation, and thermal infrared enhanced modal feature representation output by the self-attention layer after the multi-head self-attention mechanism are further mapped to a high-dimensional semantic space via a feedforward network to obtain the model. The nonlinear transformation is achieved through the GELU activation function. In order to avoid the gradient vanishing problem, a residual connection structure is adopted to maintain feature consistency, thereby improving the model stability and generalization ability.
[0035] S23-4: The high-dimensional semantic representations of the image data of each modality are obtained and input as the overall result into the dual-path adaptive Mamba module; Specifically, the visible light high-dimensional semantic representation, near-infrared high-dimensional semantic representation, and thermal infrared high-dimensional semantic representation output by the Transformer encoding module are used as the overall result and input into the dual-path adaptive Mamba module for subsequent feature decoupling and fusion.
[0036] Among them, the visible light high-dimensional semantic representation, near-infrared high-dimensional semantic representation, and thermal infrared high-dimensional semantic representation contain key region information guided by salient cues and global context information provided by random cues, providing rich semantic input for the subsequent dual-path adaptive Mamba module.
[0037] S24: The dual-path adaptive Mamba module decouples and fuses the high-dimensional semantic representations of image data from each modality, extracts modality-specific and modality-shared features from each modality of image data, and realizes deep cross-modal interaction and feature fusion to obtain a deep unified representation of multimodal image data.
[0038] The dual-path adaptive Mamba module includes a modality-specific path and a modality-shared path. The modality-specific path includes multiple first Mamba sub-modules, which are used to preserve local discriminative features using depthwise convolution. The modality-shared path includes multiple second Mamba sub-modules. These second Mamba sub-modules obtain global consistency information through state space scanning. The modality-specific path and the modality-shared path run in parallel and update the state interactively to extract the modality-specific features and modality-shared features of each modality image data, thereby realizing deep interactive modeling of multimodalities.
[0039] Specifically, such as Figure 5 As shown, it includes the following steps: S24-1: The modality-specific path models the high-dimensional semantic representation of each modality image data through the first Mamba submodule and extracts the modality-specific features of each modality image data; Specifically, the modality-specific path models the visible light high-dimensional semantic representation, near-infrared high-dimensional semantic representation, and thermal infrared high-dimensional semantic representation through the first Mamba submodule, and uses deep convolution to retain local discriminative features in order to extract visible light modality-specific features, near-infrared modality-specific features, and thermal infrared modality-specific features. The first Mamba submodule consists of depthwise separable convolutional layers, batch normalization layers, selective state space units, and SiLU activation functions. It is mainly used to extract the independent spectral characteristics and fine-grained discriminative features of each modality. This allows the modality-specific path to focus on the independent spectral characteristics and detailed differences of each modality's image data, so as to retain the unique discriminative features of each modality's image data and extract the unique features of each modality's image data.
[0040] S24-2: The modal sharing path models the high-dimensional semantic representation of each modal image data through the second Mamba submodule and extracts the modal sharing features between each modal image data; Specifically, the module shared path models the visible light high-dimensional semantic representation, near-infrared high-dimensional semantic representation, and thermal infrared high-dimensional semantic representation through the second Mamba submodule, and uses state space scanning to obtain global consistency information.
[0041] Specifically, it includes the following steps: A: The high-dimensional semantic representations of visible light, near-infrared, and thermal infrared are aggregated and summed to form a global fusion descriptor; The aggregation and summation operation can suppress modal differential responses while preserving the common activation regions of each modality, thus serving as a unified input feature for shared paths.
[0042] B: Input the global fusion descriptor into the second Mamba submodule for deep modeling. Through the state-space mechanism, the model can perform deep modeling within the global fusion descriptor. The correlation between different modal image data is captured, and feature consistency modeling at the spectral level is achieved, thereby extracting modal shared features between different modal image data.
[0043] The modality sharing features output by this path emphasize the shared attributes and consistency information between different modalities of image data.
[0044] S24-3: Perform dynamic weighted fusion of modality-shared features and modality-specific features to obtain global multimodal features; Specifically, by using learnable parameters normalized by softmax, the modality-specific features output by the modality-specific path and the modality-shared features output by the modality-shared path are dynamically weighted and fused, adaptively adjusting the contribution ratio of image data from different modalities, thereby improving cross-modal alignment and fusion capabilities while maintaining modal differences.
[0045] S24-4: Perform linear projection and normalization on global multimodal features to obtain a deep unified representation of multimodal image data, so as to realize multispectral target recognition and matching; Specifically, the global multimodal features are projected and normalized through a linear projection layer and a normalization layer to form the final unified deep representation. This unified deep representation contains both detailed information of the independent features of each modality image data and consistent semantics across modalities, providing a highly discriminative feature foundation for subsequent loss constraints and recognition matching.
[0046] S3: Based on the obtained deep unified representation, joint constraints are used to optimize the deep learning network model; Specifically, end-to-end optimization of the deep learning network model is performed based on the joint constraints of label smoothing cross-entropy loss and triplet loss to update the deep learning network model parameters, including the following steps: S31: Define label smooth cross-entropy loss to achieve classification constraints; Specifically, the label smoothing cross-entropy loss is used to guide the model to learn a stable classification boundary in the multimodal feature space. For the input sample and their corresponding real categories Output probability distribution Then, the loss function is defined as: ; in, Indicates the total number of categories. The sample belongs to the category The predicted probability is obtained by adding a small perturbation smoothing term to the true label, which can effectively reduce the problems of model overfitting and prediction bias, thereby improving the discriminativeness and robustness of cross-modal features.
[0047] S32: Define triplet loss to implement metric constraints; Triple loss is used to optimize the geometry of the embedding space, enhancing the separability between modal feature representations by maximizing inter-class distance and minimizing intra-class distance. For any anchor sample Positive samples and negative samples Its loss is defined as: ; in, Represents Euclidean distance. This is the margin hyperparameter.
[0048] Triple loss constrains the geometric structure of the feature space through metric learning, thereby improving inter-class separation and intra-class compactness, and enabling the model to maintain feature consistency across different modalities.
[0049] S33, Construct a global joint optimization objective function to achieve end-to-end training; The final optimization objective is obtained by jointly summing the losses of the Base branch and the dual-path adaptive Mamba branch:
[0050] This joint objective function operates simultaneously on the low-level feature extraction network and the high-level Mamba interaction network, achieving multi-level collaborative optimization from local features to global features. Through joint backpropagation, the model can simultaneously optimize the internal structure of modalities and the relationships between modalities, enabling the final output multimodal features to possess high discriminative power and high consistency within a unified embedding space.
[0051] S4: The multimodal image data in the validation set is processed by the optimized deep learning network model to obtain a converged deep unified representation, and multispectral target recognition and matching results are obtained based on the converged deep unified representation.
[0052] After training, based on the optimized deep learning network model, forward inference is performed on the input visible light, near-infrared, and thermal infrared images to extract the fused global multimodal feature vector. Target matching and re-identification are then achieved by calculating the Euclidean distance or cosine similarity between the features.
[0053] Experimental results on multispectral datasets show that the method of this invention outperforms existing mainstream methods in terms of recognition accuracy, recall, and cross-modal generalization, and can achieve stable multimodal target recognition in complex environments.
[0054] Example 2: This embodiment provides a multimodal target recognition system based on comprehensive prompts and Mamba deep interaction, including: The preprocessing module is used to preprocess the multimodal image data and divide the preprocessed multimodal image data into training set and validation set; the multimodal image data includes: visible light image data, near-infrared image data and thermal infrared image data.
[0055] Deep learning network models are used to process preprocessed multimodal image data to obtain a deep unified representation of the multimodal image data. The deep learning network model includes: a pre-trained CLIP visual coding model, which is used to embed features into preprocessed multimodal image data to obtain basic modal feature representations of each modal image data; The salient-random suggestion aggregation module is used to generate enhanced modal feature representations for each modal image data based on the basic modal feature representations of each modal image data. The Transformer encoding module is used to enhance modal feature representation based on image data of each modality and obtain a unified high-dimensional semantic representation across modalities. The dual-path adaptive Mamba module is used to decouple and fuse the high-dimensional semantic representations of image data from various modalities, extract modality-specific features and modality-shared features from each modality of image data, and realize deep cross-modal interaction and feature fusion to obtain a deep unified representation of multimodal image data. The training module trains the deep learning network model using the training set, optimizes the parameters of the deep learning network model, and enables it to output a deep unified representation with strong discriminative power for the input multimodal image data. The optimization module optimizes the deep learning network model using joint constraints based on the obtained deep unified representation. The target matching and recognition module processes multimodal image data in the validation set through an optimized deep learning network model to obtain a converged deep unified representation, and obtains multispectral target recognition and matching results based on the converged deep unified representation.
[0056] Example 3: The present invention provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the computer-readable storage medium is located to execute the multimodal target recognition method based on comprehensive prompts and Mamba deep interaction described in Embodiment 1 above.
[0057] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.
[0058] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can... This can be achieved through other means. For example, the system embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.
[0059] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0060] Additionally, it should be noted that the flowcharts in the accompanying drawings illustrate methods according to embodiments of this disclosure. In the descriptions corresponding to the flowcharts or block diagrams in the drawings, the operations or steps corresponding to different blocks may occur in a different order than disclosed in the description; sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, or sometimes in reverse order, depending on the function involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.
[0061] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A multimodal target recognition method based on comprehensive prompts and Mamba deep interaction, characterized in that, Includes the following steps: Step 1: Preprocess the multimodal image data and divide the preprocessed multimodal image data into a training set and a validation set. The multimodal image data includes: visible light image data, near-infrared image data, and thermal infrared image data. Step 2: Train the deep learning network model using the training set, optimize the parameters of the deep learning network model, so that it can output a deep unified representation with strong discriminative power for the input multimodal image data; Step 3: Perform end-to-end optimization of the deep learning network model based on the joint constraints of label smoothing cross-entropy loss and triplet loss to update the parameters of the deep learning network model; Step 4: The multimodal image data in the validation set is processed using the optimized deep learning network model to obtain a converged deep unified representation. Based on the converged deep unified representation, the multispectral target recognition and matching results are obtained. Step two includes: inputting the preprocessed multimodal image data into a pre-trained CLIP visual coding model for feature embedding to obtain the basic modal feature representation of each modal image data; Based on the basic modal feature representations of each modal image data, an enhanced modal feature representation of each modal image data is generated through a salient-random cue aggregation module; The enhanced modal feature representations of each modality image data are input into the Transformer encoding module to obtain a unified high-dimensional semantic representation across modalities; By using a dual-path adaptive Mamba module, the high-dimensional semantic representations of image data from various modalities are decoupled and fused, and modality-specific and modality-shared features of each modality are extracted to achieve deep cross-modal interaction and feature fusion, thereby obtaining a deep unified representation of multimodal image data.
2. The multimodal target recognition method based on comprehensive prompts and Mamba deep interaction according to claim 1, characterized in that, The step of inputting the preprocessed multimodal image data into a pre-trained CLIP visual coding model for feature embedding to obtain the basic modal feature representations of each modality of image data includes: The pre-trained CLIP visual coding model performs region segmentation and feature extraction on the preprocessed input image data of each modality to obtain patch-level feature sequences that represent the local semantic information of each modality image data; Linear mapping and embedding are performed on the patch-level feature sequences representing local semantic information of each modality image data to form a basic modal feature representation of each modality image data with a unified dimension.
3. The multimodal target recognition method based on comprehensive prompts and Mamba deep interaction according to claim 1, characterized in that, The basic modal feature representation based on each modal image data, through the salient-random cue aggregation module, generates enhanced modal feature representations for each modal image data, including: Based on the L2 norm of the basic modal feature representation of each modal image data, salient features are selected; Random prompts are generated by randomly selecting several basic modal feature representations from the basic modal feature representations of each modal image data. By concatenating random cues with prominent cues, a comprehensive cues with modality invariance are generated; The generated comprehensive prompts are concatenated with the basic modal feature representations of each modal image data to obtain the enhanced modal feature representations of each modal image data.
4. The multimodal target recognition method based on comprehensive prompts and Mamba deep interaction according to claim 1, characterized in that, The step of inputting the enhanced modal feature representations of each modality image data into the Transformer encoding module to obtain a unified high-dimensional semantic representation across modalities includes: Linear projection and normalization were performed on the image data for each modality; A multi-head self-attention mechanism is performed on the normalized image data of each modality to achieve feature interaction and semantic alignment within and between the image data of each modality; The image data of each modality after the multi-head self-attention mechanism has been executed is fed into the feedforward network and combined with residual connections to obtain the high-dimensional semantic representation of the image data of each modality.
5. The multimodal target recognition method based on comprehensive prompts and Mamba deep interaction according to claim 1, characterized in that, The dual-path adaptive Mamba module includes modality-specific paths and modality-shared paths; The modality-specific path includes multiple first Mamba sub-modules, which are used to preserve local discriminative features using depthwise convolution; The modal sharing path includes multiple second Mamba submodules; these second Mamba submodules obtain global consistency information through state space scanning.
6. The multimodal target recognition method based on comprehensive prompts and Mamba deep interaction according to claim 5, characterized in that, The process of using a dual-path adaptive Mamba module to decouple and fuse the high-dimensional semantic representations of image data from various modalities, extracting modality-specific and modality-shared features from each modality's image data, and achieving deep cross-modal interaction and feature fusion to obtain a deep unified representation of multimodal image data includes: The modality-specific path models the high-dimensional semantic representation of each modality image data through the first Mamba submodule and extracts the modality-specific features of each modality image data; The modality sharing path models the high-dimensional semantic representation of each modality image data through the second Mamba submodule and extracts the modality sharing features between each modality image data. Dynamic weighted fusion is performed on modality-shared features and modality-specific features to obtain global multimodal features; Linear projection and normalization are performed on global multimodal features to obtain a deep unified representation of multimodal image data.
7. The multimodal target recognition method based on comprehensive prompts and Mamba deep interaction according to claim 1, characterized in that, Step three includes: Define a label-smooth cross-entropy loss to achieve classification constraints; Define triplet loss to achieve metric constraints; A global joint optimization objective function is constructed to achieve end-to-end training.
8. A multimodal target recognition system based on comprehensive prompts and Mamba deep interaction, characterized in that, include: The preprocessing module is used to preprocess the multimodal image data and divide the preprocessed multimodal image data into training set and validation set; Deep learning network models are used to process preprocessed multimodal image data to obtain a deep unified representation of the multimodal image data. The deep learning network model is trained using a training set, and its parameters are optimized so that it can output a deep unified representation with strong discriminative power for the input multimodal image data. The optimization module optimizes the deep learning network model using joint constraints based on the obtained deep unified representation. The target matching and recognition module processes multimodal image data in the validation set through an optimized deep learning network model to obtain a converged deep unified representation, and obtains multispectral target recognition and matching results based on the converged deep unified representation. The deep learning network model includes: a pre-trained CLIP visual coding model, used to embed features into preprocessed multimodal image data to obtain basic modal feature representations of each modal image data; The salient-random suggestion aggregation module is used to generate enhanced modal feature representations for each modal image data based on the basic modal feature representations of each modal image data. The Transformer encoding module is used to enhance modal feature representation based on image data of each modality and obtain a unified high-dimensional semantic representation across modalities. The dual-path adaptive Mamba module is used to decouple and fuse features of high-dimensional semantic representations of image data from various modalities, extract modality-specific features and modality-shared features of each modality of image data, realize deep cross-modal interaction and feature fusion, and obtain a deep unified representation of multimodal image data.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the multimodal target recognition method based on comprehensive prompts and Mamba deep interaction as described in any one of claims 1 to 7.