Multi-modal target re-identification learning method, system and device and readable medium
By truncating the visual Mamba backbone network and densely connected Mamba module, the problems of weak feature discrimination and high computational complexity in multimodal target re-identification are solved, and efficient multimodal feature extraction and robustness improvement are achieved.
Patent Information
- Application Number
- CN202410968126.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing multimodal target recognition method has shortcomings in feature discrimination and computational complexity, especially the CNN-based method has weak feature distinction, and the Transformer-based method has high computational complexity and ignores the locality of different image modes. detail.
The truncated visual Mamba backbone network (TVMB) is used to combine densely connected Mamba modules and consistent modal fusion components, and to integrate cross-modal depth features through dense connections, reduce computing complexity and improve feature robustness.
It realizes the extraction of multimodal features with high discrimination at low computational complexity, significantly improving the robustness and discrimination of features, surpassing the performance of most advanced methods.
Smart Images

Figure CN118982818A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of multimodal target re-identification, and in particular relates to a multimodal target re-identification learning method, system, device and readable medium. Background Art
[0002] Object Re-Identification (ReID) aims to identify the same object from different camera views. Due to its wide range of practical applications, object re-identification (ReID) has made remarkable progress in recent years. Traditional object ReID mainly focuses on extracting discriminative information from easily acquired RGB images. However, complex imaging conditions, such as darkness, strong illumination, and low image resolution, can severely affect the quality of RGB images. Fortunately, images of other modalities (e.g., near-infrared (NIR)), thermal infrared (TIR)) show great potential in overcoming these challenges. By integrating complementary information from multimodal images, it can provide a more robust representation. Therefore, multimodal object ReID has received increasing attention.
[0003] In order to fuse heterogeneous information from different image modalities, a heterogeneous collaborative sensing multi-stream network (HAMNet) is used to aggregate different image spectral features. A progressive fusion network (PFNet) is used to gradually aggregate multimodal features from local to global. Three learning strategies are used to integrate modality-specific knowledge in multimodal person ReID. A dynamic enhancement network (DENet) is introduced to address the modality missing problem. A generative and attentive fusion network (GAFNet) is also used to fuse heterogeneous information from a generative perspective. Recently, with the powerful generalization ability of Transformers, a progressive hybrid transform (PHT) is introduced to balance the useful information from different image modalities. By directly cascading the features of modality-specific Transformers, a strong baseline named UniCat is provided. A cyclic token replacement mechanism for robust multimodal object ReID is adopted. Through test-time training, the useful information in the test data is explored, as well as the selection of diverse features and the minimization of the impact of background noise are explored.
[0004] Although the above methods have achieved good results, they still have certain limitations. For CNN-based methods, their performance is limited by the limited receptive field, resulting in weak feature discrimination. For Transformer-based methods, the endogenous attention mechanism leads to high computational complexity. In addition, the lack of inductive bias causes the typical Transformer to ignore certain local details of different image modalities. The current methods still face some challenges that need to be addressed, such as weak feature discrimination and high computational complexity. Therefore, it is necessary to develop a new multimodal object re-identification learning method, system, device and readable medium to solve the existing problems. Summary of the invention
[0005] The purpose of the present invention is to provide a multimodal target re-identification learning method, system, device and readable medium to solve the problem of weak feature discrimination.
[0006] To achieve the above object, the present invention provides the following technical solution: a multimodal target re-identification learning method, comprising:
[0007] Extract multimodal representations from RGB, NIR, and TIR modality images;
[0008] The multimodal representation is fused with shallow and deep features; deep features across modalities at the same spatial position are integrated to generate multimodal features.
[0009] Preferably, extracting the multimodal representation from the RGB modality image comprises:
[0010] The input RGB modality image is represented as I R ;
[0011] Where H, W and C represent the height, width and number of channels of the image respectively;
[0012] The RGB image is sent to the Stem module with a convolutional layer to extract the initial features, which are recorded as
[0013] Will The global information and the local information are preferably extracted by processing through a plurality of visual state space blocks, wherein the visual state space blocks include a residual connection:
[0014]
[0015] in represents the output of the lth visual state space block.
[0016] Preferably, the multimodal representation fusing shallow and deep features comprises:
[0017] Processing is performed through a densely connected Mamba module, which includes: Dense connection formula:
[0018]
[0019] Wherein, in formula (2), l is the lth visual state space block in the densely connected Mamba module, l = 1, 2, ..., d, d is the number of visual state space blocks in the densely connected Mamba module; m represents the accumulated variable;
[0020] When l = 1, that is, after a visual state space block, Here Refers to the feature input of the densely connected Mamba module;
[0021] In formula (3) Represents the features and averages from the 1st to the lth visual state space blocks;
[0022] After DCMM dense connection, the feature is finally obtained
[0023] Preferably, the extracting the multimodal representation from the NIR and TIR modal images comprises: representing the NIR and TIR multimodal images as I N and I T ; Obtain feature F in NIR and TIR modes N and F T The steps are the same as those for extracting multimodal representations from RGB modality.
[0024] Preferably, the integration of cross-modal deep features at the same spatial position includes: using a linear transformation layer to reduce F R 、F N and F T The channel dimension of is as shown in formula (4);
[0025]
[0026] Among them, F R represents the image features of the RGB image after truncated visual Mamba backbone network, F N represents the image features of the NIR image after truncating the visual Mamba backbone network, F T Represents the image features of the TIR image after truncating the visual Mamba backbone network;
[0027] Use the consistency loss L C The characteristics and To align:
[0028]
[0029] In formula (5): N P is the number of tokens. Formula (6), Formula (7) and Formula (8) represent feature alignment operations. After alignment, concatenation is performed along the channel dimension to obtain multimodal features. The formula is as follows:
[0030]
[0031] Where [·] represents the concatenation operation;
[0032] Will Input into a visual state-space block with K layers:
[0033]
[0034] in Represents the output of the kth visual state space block, and uses global average pooling to obtain the fusion feature
[0035]
[0036] Preferably, the truncated visual Mamba backbone network and the deep features between the cross-modal features integrating the same spatial position are both subjected to the label smoothed cross entropy loss L ID and triplet loss L Triplet Supervision:
[0037] L TVMB =L ID (F tvmb )+L Triplet (F tvmb ) (12)
[0038] L CMF =L ID (F cmf )+L Triplet (F cmf ) (13)
[0039] F tvmb =[GAP(F R ),GAP(F N ),GAP(F T )] (14)
[0040] Among them, L ID () represents the label smoothed cross entropy loss function, L Triplet () represents the triple loss function, F tvmb Indicates F R 、F N 、FT The concatenated features, F cmf It represents the features after the CMF module operation, GAP() represents the global average pooling function;
[0041] The overall objective function is defined as:
[0042] L=L TVMB +L CMF +λL C (15)
[0043] Where: λ is used to balance L C Hyperparameters of .
[0044] The present invention further provides a multimodal object re-identification learning system, comprising: a truncated visual Mamba backbone network for extracting robust multimodal representations from different modalities;
[0045] Densely connected Mamba modules to integrate information from shallow and deep features;
[0046] The consistent modality fusion component is used to integrate the deep features across modalities at the same spatial location.
[0047] The present invention further provides a multimodal target re-identification learning device, comprising:
[0048] a memory for storing non-transitory computer-readable instructions; and
[0049] A processor is used to run the computer-readable instructions, so that when the computer-readable instructions are executed by the processor, the multimodal target re-identification learning method is implemented.
[0050] The present invention further provides a computer-readable storage medium for storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the computer executes the above-mentioned multimodal target re-identification learning method.
[0051] Technical effects and advantages of the present invention: The multimodal target re-identification learning method, system, device and readable medium, the framework consists of three key components: Truncated Vision Mamba Backbone (TVMB), Densely Connected Mamba Module (DCMM) and Consistent Modality Fusion (CMF); firstly, Vision Mamba is truncated and our TVMB is constructed to efficiently capture global context and local details with lower computational complexity; then, in order to further improve the feature discrimination, the inter-modal information is fused with shallow and deep features through the Densely Connected Mamba Module (DCMM); Consistent Modality Fusion (CMF) is used to combine the inter-modal information with the shallow and deep features; Abstract: In this paper, a novel multimodal image recognition framework based on Mamba is proposed, which can significantly improve the robustness of features. A large number of experiments on three public benchmark test sets show that the present invention achieves better performance than most state-of-the-art methods and integrates Mamba into multimodal object ReID for the first time. A three-stage VMamba (TVMB) is used to extract robust multimodal representations. TVMB can effectively capture global contextual information and local details while maintaining low computational complexity, and seamlessly integrates cross-modal information of shallow and deep features through densely connected Mamba modules (DCMM). Consistent Modal Fusion (CMF) is used to fuse deep features from different modalities. Using well-aligned multimodal images, CMF improves the modality aggregation granularity, thereby enhancing the discriminability of features. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 It is a schematic diagram of the framework of the present invention;
[0053] Figure 2 A visualization diagram of the feature distribution using t-SNE for the present invention;
[0054] Figure 3 This is a detailed diagram of the CMF of the present invention;
[0055] Figure 4 This is a schematic diagram of the VSS block of the present invention;
[0056] Figure 5 This is a detailed diagram of the DCMM of the present invention. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0058] The present invention provides Figure 1 A new feature learning framework MambaReID is presented in this paper for multimodal object re-identification task, which has strong feature discrimination and low computational complexity. Specifically, the framework consists of three key components: Truncated Vision Mamba Backbone (TVMB), Densely Connected Mamba Module (DCMM) and Consistent Modality Fusion (CMF). By truncating Vision Mamba, TVMB can effectively capture global context and local detail information with low computational complexity. To further improve feature discrimination, DCMM fully fuses inter-modal information with shallow and deep features through dense connections. Consistent Modality Fusion CMF realizes multimodal fusion, thereby significantly improving the robustness of features. Through the proposed module, more discriminative features can be obtained for multimodal object re-identification. Extensive experiments on three public benchmark datasets verify the effectiveness of the proposed method.
[0059] TVMB is adopted as the backbone network, aiming to extract robust three-modal representations from RGB, NIR and TIR modality images. DCMM is introduced to the last level of TVMB to further enhance the feature representation capability of dense connections. Finally, CMF is used to enhance the multimodal feature aggregation granularity in the case of multimodal image alignment.
[0060] Images of RGB, NIR, and TIR modalities are fed into the backbone network TVMB. With the help of lightweight TVMB, robust multimodal representations are extracted from different modalities. In the final stage of TVMB, DCMM is used to further integrate information from shallow and deep features. Finally, CMF is used to enhance the granularity of modality aggregation at the same spatial location in the case of multimodal image alignment. With the proposed module, our MambaReID is able to generate more discriminative multimodal features with lower computational complexity.
[0061] In order to extract discriminative features with lower computational complexity, Visual Mamba is used as the backbone. However, the last stage of Visual Mamba introduces very redundant computational costs. Therefore, a truncated structure is adopted to directly remove the last downsampling layer and the last stage.
[0062] Specifically, the input RGB, NIR and TIR multimodal images are represented as I R , I N and I T , I R Represents an image tensor. Without loss of generality, we take RGB image as an example. Where H, W and C represent the height, width and number of channels of the image respectively. First, the RGB image is sent to the Stem module with a convolutional layer to extract the initial features, which are recorded as Then, It is processed through multiple visual state space (VSS) blocks to extract global and local information. VSS blocks such as Figure 4 As shown. In order to improve the learning ability, the following residual connection is introduced:
[0063]
[0064] in represents the output of the lth VSS block. First, the feature is sent to the next stage for further processing. Similarly, features can be obtained Finally, through the last level of TVMB, the features are obtained.
[0065] Densely Connected Mamba Module (DCMM), In practical applications, local details play an important role in multimodal object ReID. Simple stacking of VSS blocks may extract limited local information. Inspired by the feature reuse of dense connections, the densely connected Mamba module (Densely Connected Mamba Module, DCMM) is used to further improve the feature representation capability. Dense connections have been widely used in various computer vision tasks. However, directly introducing dense connections at each stage in TVMB will lead to a significant increase in complexity. Dense Mamba is introduced to the last level of TVMB. Figure 5 As shown, specifically, we add all previous features to the current features, and DCMM can be expressed as formulas (2) and (3);
[0066]
[0067] In formula (2), l is the lth VSS block in the DCMM module, l = 1, 2, ..., d, d is the number of VSS blocks in the DCMM module, that is, there are d VSS blocks in the DCMM. In particular, when l = 1, that is, after one VSS block, Here It refers to the characteristic input of the DCMM module, that is, in the overall framework diagram, the H / 8*W / 8*C3 marked corresponds to the output characteristics of the second stage.
[0068] Through dense connections, the information of shallow and deep features of a single modality image can be fully integrated, which can effectively enhance the discriminability of features to obtain better object perception. Similarly, features F can be obtained from other modes N and F T . Represents the characteristics and average value from the 1st to the lth VSS block.
[0069] Consistent Modal Fusion (CMF)
[0070] In the aligned multimodal images, in order to fully explore the complementary information between different image modalities, consistent modal fusion (CMF) is used to integrate the deep features between the modalities at the same spatial position, such as Figure 3 As shown, a linear transformation layer is first used to reduce F R 、F N and F T The channel dimension of is as shown in formula (4);
[0071]
[0072] F R represents the image features of the RGB image after TVMB, F N 、F T The features of the other two modal images obtained after passing through the same network, that is, the features obtained after passing through the DCMM module;
[0073] Then, using the consistency loss L C The characteristics and To align:
[0074]
[0075] Where: N Pis the number of tokens. Formula (6), Formula (7) and Formula (8) represent feature alignment operations. After alignment, concatenation is performed along the channel dimension to obtain the following multimodal features. The formula is as follows:
[0076]
[0077] Where [·] represents a concatenation operation.
[0078] Then, Input into a VSS block with K layers:
[0079]
[0080] in Represents the output of the kth VSS block. The fusion feature is obtained by using global average pooling (GAP)
[0081]
[0082] Utilize fully fused multimodal features to obtain more discriminative representations.
[0083] Objective Function
[0084] For model training, our objective function consists of three parts. For TVMB backbone and CMF, they are both subject to label smoothed cross entropy loss L ID and triplet loss L Triplet Supervision:
[0085] L TVMB =L ID (F tvmb )+L Triplet (F tvmb ) (12)
[0086] L CMF =L ID (F cmf )+L Triplet (F cmf ) (13)
[0087] F tvmb =[GAP(F R ),GAP(F N ),GAP(F T )] (14)
[0088] Therefore, the overall objective function is defined as:
[0089] L=L TVMB +L CMF +λL C (15)
[0090] Where λ is used to balance L C The hyperparameters of L are L. Since the VSS block has long-range modeling ability, these features can capture the global context. By minimizing L, MambaReID can generate more discriminative multimodal features with lower computational complexity.
[0091] In formulas (12)-(14), L ID () represents the label smoothed cross entropy loss function, L Triplet () represents the triple loss function;
[0092] F tvmb Indicates F R 、F N 、F T First, the features are pooled separately, and then the three are concatenated together. cmf It represents the features after the CMF module operation. GAP() means the global average pooling function. The role of TVMB is to fully extract the image features of different modal images of the same object under the premise of low computational overhead. The role of DCMM is to improve the feature representation ability of TVMB output features without significantly increasing the computational overhead. The role of CMF is to enhance the granularity of modal aggregation at the same spatial position.
[0093] Datasets and evaluation metrics:
[0094] To comprehensively evaluate the performance of our scheme, three multimodal object ReID benchmarks are employed. More specifically, RGBNT201 is a multimodal pedestrian ReID dataset containing three modalities: RGB, NIR, and TIR. RGBNT100 is a large-scale multimodal vehicle ReID dataset, while MSVR310 is a small-scale dataset with complex scenes. For evaluation, mean average precision (mAP) and cumulative matching features (CMC) are used on Rank-K (K=1,5,10). In addition, the trainable parameters and FLOPs for complexity analysis are reported.
[0095] The proposed model is implemented using the PyTorch toolbox using a single NVIDIA A100 GPU. The image size is 256×128 for RGBNT201 and 128×256 for RGBNT100 and MSVR310. During training, random horizontal flipping, cropping, and erasing are used for data augmentation. The mini-batch size is set to 64, containing 8 randomly selected identities, and 8 images are sampled for each identity. The Stochastic Gradient Descent (SGD) optimizer with momentum of 0.9 and weight decay of 0.0001 is used. The learning rate is initialized to 0.01, following a warm-up strategy with cosine decay. λ is set to 1, d = 1,…,14. During testing, F tvmb and F cmf Combined for object retrieval.
[0096] Comparison with existing state-of-the-art methods
[0097] Tables 1 and 2 show the performance comparison with other methods. The best and second best methods are highlighted in bold and underlined, respectively. The symbol * indicates the Transformer-based method, and the symbol represents the Mamba-based method. The others are CNN-based methods.
[0098] Multimodal Person ReID. As can be observed in Table 1. CNN-based methods show poor performance due to limited receptive field. Transformers, HTT and EDITOR achieve superior performance. The best mAP of 72.2% and Rank-1 of 76.0% are achieved in this scheme, which verifies the effectiveness of this scheme.
[0099] Multimodal vehicle ReID. Table 2 shows the comparison results of RGBNT100 and MSVR310. In general, the Transformer-based method shows superior performance. A competitive Rank-1 of 93.4% was achieved on RGBNT100. Meanwhile, on MSVR310, 46.0% Rank-1 was achieved, surpassing EDITOR by 2.9%. These results demonstrate the effectiveness of this scheme.
[0100] Table 1. Performance comparison on the pedestrian dataset RGBNT201
[0101]
[0102]
[0103] Table 2. Performance comparison on vehicle datasets RGBNT100 and MSVR310
[0104]
[0105]
[0106] Ablation experiment
[0107] Impact of key components. As shown in Table 3, the impact of key components is demonstrated. Model A is the baseline. Under CMF, Model B achieves a mAP of 68.03%, exceeding Model A by 4.22%. In DCMM, Model C achieves a mAP of 68.07%, exceeding Model A by 4.26%. Our final method achieves the best mAP of 72.20%. In terms of complexity, Model B and Model C only introduce small computational overhead. These results fully verify the effectiveness of our components.
[0108] Table 3. Performance comparison of key components
[0109]
[0110] Parameter Analysis. Table 4 shows the trainable parameters of different methods. Our method has only 59.47M parameters, which significantly reduces the memory requirement.
[0111] Table 4. Parameter comparison of different methods
[0112]
[0113]
[0114] Effect of different backbones. Table 5 shows the performance of different backbone networks. ViT achieves better results than directly using Vision Mamba. However, ViT has a higher computational complexity. Using only 63.96% of ViT’s parameters and 79.22% of FLOPs, our scheme achieves 63.81% mAP, exceeding ViT’s 0.63%. These results verify the effectiveness of the backbone network.
[0115] Table 5. Performance comparison of different backbone networks
[0116]
[0117] Effect of different depths of CMF. Table 6 shows the effects of different depths of CMF. As the depth increases, the performance gradually decreases. This indicates that deeper CMF may introduce more irrelevant information, resulting in performance degradation. Therefore, the depth is set to 1.
[0118] Table 6. Comparison of CMF at different depths
[0119]
[0120]
[0121] Visual Analytics
[0122] Figure 2 Visualize the feature distribution of different modules on RGBNT201. Figure 2 (a) shows the original Vision Mamba features, which are separated into the same ID, resulting in poor discrimination. Figure 2 As shown in (b), TVMB makes the features within an ID grouped more tightly. Through DCMM and CMF, our method obtains a more compact feature distribution. These visualizations provide strong evidence for the effectiveness of our different modules.
[0123] A new feature learning framework MambaReID is proposed for multimodal object ReID. A Mamba-based backbone network is used to extract robust multimodal representations with low computational overhead. In addition, a densely connected Mamba module is introduced to further enhance the discriminability of features. Consistent Modality Fusion (CMF) is used to fully aggregate multimodal information. Extensive experiments on three public benchmarks demonstrate the effectiveness of this scheme.
[0124] The present invention further provides a multimodal object re-identification learning system, comprising: a truncated visual Mamba backbone network for extracting robust multimodal representations from different modalities;
[0125] Densely connected Mamba modules to integrate information from shallow and deep features;
[0126] The consistent modality fusion component is used to integrate the deep features across modalities at the same spatial location.
[0127] The present invention further provides a multimodal target re-identification learning device, comprising:
[0128] a memory for storing non-transitory computer-readable instructions; and
[0129] A processor is used to run the computer-readable instructions, so that when the computer-readable instructions are executed by the processor, the multimodal target re-identification learning method is implemented.
[0130] The present invention further provides a computer-readable storage medium for storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the computer executes the above-mentioned multimodal target re-identification learning method.
[0131] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0132] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0133] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0135] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0136] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
[0137] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A multimodal object re-identification learning method, characterized by: include: Extract multimodal representations from RGB, NIR, and TIR modality images; The multimodal representation described herein fuses shallow and deep features; Integrate the deep features across modalities at the same spatial position to generate multimodal features.
2. The multimodal object re-identification learning method according to claim 1, characterized in that: The extracting of multimodal representation from the RGB modality image comprises: The input RGB modality image is represented as I R ; Where H, W and C represent the height, width and number of channels of the image respectively; The RGB image is sent to the Stem module with a convolutional layer to extract the initial features, which are recorded as Will It is processed through multiple visual state space blocks to extract global and local information.
3. The multimodal object re-identification learning method according to claim 2, characterized in that: The visual state space block includes a residual connection: in represents the output of the lth visual state space block.
4. The multimodal object re-identification learning method according to claim 1, characterized in that: The multimodal representation combines shallow and deep features by: Dense connection formula: Wherein, in formula (2), l is the lth visual state space block in the densely connected Mamba module, l = 1, 2, ..., d, d is the number of visual state space blocks in the densely connected Mamba module; m represents the accumulated variable; When l = 1, that is, after a visual state space block, Here Refers to the feature input of the densely connected Mamba module; In formula (3) Represents the features and averages from the 1st to the lth visual state space blocks; After dense connection, the features are obtained 5. The multimodal object re-identification learning method according to claim 1, characterized in that: The extracting multimodal representation from the NIR and TIR modal images comprises: representing the NIR and TIR multimodal images as I N and I T ; Obtain feature F in NIR and TIR modes N and F T The steps are the same as those for extracting multimodal representations from RGB modality.
6. The multimodal object re-identification learning method according to claim 1, characterized in that: The integration of cross-modal deep features at the same spatial position includes: using a linear transformation layer to reduce F R 、F N and F T The channel dimension of is as shown in formula (4); Among them, F R represents the image features of the RGB image after truncated visual Mamba backbone network, F N represents the image features of the NIR image after truncating the visual Mamba backbone network, F T Represents the image features of the TIR image after truncating the visual Mamba backbone network; Use the consistency loss L C The characteristics and To align: In formula (5): N P is the number of tokens. Formula (6), Formula (7) and Formula (8) represent feature alignment operations. After alignment, concatenation is performed along the channel dimension to obtain multimodal features. The formula is as follows: Where [·] represents the concatenation operation; Will Input into a visual state-space block with K layers: in Represents the output of the kth visual state space block, and uses global average pooling to obtain the fusion feature 7. A multimodal object re-identification learning method according to claim 6, characterized in that: The truncated visual Mamba backbone network and the deep features integrating the same spatial position across modalities are both subject to the label smoothed cross entropy loss L ID and triplet loss L Triplet Supervision: L TVMB =L ID (F tvmb )+L Triplet (F tvmb ) (12) L CMF =L ID (F cmf )+L Triplet (F cmf ) (13) F tvmb =[GAP(F R ),GAP(F N ),GAP(F T )] (14) Among them, L ID () represents the label smoothed cross entropy loss function, L Triplet () represents the triple loss function, F tvmb Indicates F R 、F N 、F T The concatenated features, F cmf It represents the features after the CMF module operation, GAP() represents the global average pooling function; The overall objective function is defined as: L=L TVMB +L CMF +λL C (15) Where: λ is used to balance L C Hyperparameters of .
8. The multimodal object re-identification learning system according to claim 1, characterized in that: include: Truncated Visual Mamba Backbone for extracting robust multimodal representations from different modalities; Densely connected Mamba modules to integrate information from shallow and deep features; The consistent modality fusion component is used to integrate the deep features across modalities at the same spatial location.
9. The multimodal object re-identification learning device according to claim 1, characterized in that: include: a memory for storing non-transitory computer-readable instructions; as well as A processor, configured to run the computer-readable instructions, so that when the computer-readable instructions are executed by the processor, the multimodal target re-identification learning method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium for storing non-transitory computer-readable instructions, which, when executed by a computer, enables the computer to execute the multimodal target re-identification learning method described in any one of claims 1 to 7.
Citation Information
Patent Citations
Cross-modal pedestrian re-identification method and device based on intermediate modal and representation learning
CN116311384A
Complex scene personnel re-identification method based on multi-modal fusion
CN118334375A
Cross-modal person re-identification method and device
WO2022027986A1
Cited By
Target re-identification method and device, medium and product
CN119380006A
Traditional Chinese medicinal material identification method based on RGB image and near infrared spectrum
CN121121369A
