Multi-modal vehicle re-identification method and system based on flow generation multi-expert fusion

Abstract

The application relates to a multi-modal vehicle re-identification method and system based on flow generation type multi-expert fusion, which comprises the following steps: acquiring a vehicle image and extracting input features, and classifying available modalities and missing modalities according to the availability of modalities; modeling a multi-modal joint latent distribution based on normalized flow, inferring the conditional distribution parameters of the missing modalities in the latent space under the condition of the available modalities, and generating semantic consistent missing modality features through reversible transformation; combining missing mode perception routing and modality level uncertainty weighting strategies to perform semantic enhancement and deviation compensation on the restored missing modality features, and obtaining vehicle identity embedding representation; and realizing vehicle re-identification based on the vehicle identity embedding representation. Compared with the prior art, the application has the advantages of effectively processing various missing modality conditions and maintaining the complementary relationship between modalities.

Description

A multimodal vehicle re-identification method and system based on stream generation and multi-expert fusion. Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multimodal vehicle re-identification method and system based on stream-generative multi-expert fusion. Background Technology

[0002] Vehicle Re-Identification (VehReID) is a core technology in intelligent transportation systems, designed to match and track the same vehicle across different camera perspectives. This technology has significant application value in areas such as traffic monitoring, parking management, and urban security.

[0003] Early vehicle re-identification methods primarily relied on RGB (visible light) images, using feature extractors to extract visual attributes of vehicle appearance, such as color, shape, and brand logos, for matching. However, vehicle re-identification methods relying solely on RGB modalities have significant limitations. First, under low-light conditions, the quality of RGB images deteriorates severely, making feature extraction difficult. Second, in adverse weather conditions, the contrast and sharpness of RGB images decrease, affecting recognition accuracy. Third, in strong light or backlight environments, RGB images are prone to overexposure or underexposure, losing crucial details. Finally, RGB images are insensitive to the vehicle's internal structure and thermal distribution information, making it difficult to distinguish between vehicles that appear similar but are fundamentally different.

[0004] To overcome these limitations of RGB modality, researchers have begun exploring multispectral vehicle re-identification techniques, introducing additional modal data such as near-infrared (NIR) and thermal infrared (TIR). Current mainstream methods for handling missing modalities primarily focus on pixel-level and feature-level generation, but still suffer from the following limitations:

[0005] 1. Pixel-level generation methods are difficult to learn and cannot dynamically respond to modal missingness during the inference stage. To address the missingness problem, some methods attempt to generate complete images in pixel space;

[0006] 2. Feature-level compensation methods have coarse modeling and lack refined dynamic adaptability.

[0007] In summary, existing methods are either hampered by the infeasibility of high-dimensional modeling and deployment at the pixel level, or limited to low-precision approximations and static strategies based on feature-level compensation. Both lack a unified, end-to-end, inference-time adaptive modality recovery and fusion framework, making it difficult to maintain robust recognition performance under arbitrary modality loss conditions. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a multimodal vehicle re-identification method and system based on stream generation multi-expert fusion, aiming to improve the robustness and accuracy of vehicle re-identification by effectively handling modality loss.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] One aspect of the present invention provides a multimodal vehicle re-identification method based on stream-generated multi-expert fusion, comprising the following steps:

[0011] Acquire vehicle images and extract input features, classifying them into available modes and missing modes based on their availability;

[0012] Based on normalized flow, the joint latent distribution of multimodal modes is modeled. Using available modes as conditions, the conditional distribution parameters of missing modes in the latent space are inferred, and semantically consistent missing mode features are generated through invertible transformation.

[0013] By combining missing mode-aware routing and modal-level uncertainty weighting strategies, semantic enhancement and bias compensation are performed on the recovered missing modal features to obtain the vehicle identity embedding representation;

[0014] Vehicle re-identification is achieved based on the vehicle identity embedding representation.

[0015] As a preferred technical solution, the extraction of input features is implemented based on a multimodal feature extraction network, including:

[0016] Based on the modality type of the vehicle image, features are extracted using the corresponding feature extractor and then subjected to global average pooling to obtain the input features.

[0017] As a preferred technical solution, the process of generating semantically consistent missing modality features through reversible transformation includes the following steps:

[0018] Using a flow model based on multi-layer conditional affine coupling, the spliced ​​vector of available modes is forward transformed and mapped to the latent space. In each layer of flow, the parameters of the scaling function and translation function are generated by a conditional network, and the input of the conditional network is the spliced ​​vector of available modes.

[0019] Using the concatenated vector of available modalities as input to a lightweight neural network, the output is the conditional mean of the missing modal features;

[0020] The conditional mean is concatenated with the concatenated vector of available modes to obtain a latent vector. This latent vector is then mapped back to the original feature space through the inverse transformation of the flow model to obtain the recovered missing mode features.

[0021] As a preferred technical solution, the lightweight neural network is trained in conjunction with the flow model.

[0022] As a preferred technical solution, the process of obtaining the vehicle identity embedded representation includes the following steps:

[0023] For the features of the available modalities and the features of the recovered missing modalities, calculate the self-attention and cross-modal attention enhancement features respectively;

[0024] Based on the self-attention and cross-modal attention enhancement features, the modal-level relative gain entropy of each modality is calculated;

[0025] Based on the available and missing modalities, match the corresponding missing modal pattern, activate the corresponding expert group, and realize missing pattern-aware routing.

[0026] Within the activated expert group, expert weights are calculated based on the modal-level relative gain entropy corresponding to the modality.

[0027] Based on the expert weights, compensated enhancement features are generated to obtain the vehicle identity embedding representation, thereby achieving modal-level uncertainty weighting.

[0028] As a preferred technical solution, the calculation process of the modal level relative gain entropy includes the following steps:

[0029] Calculate the sum of the exponential responses of each attention branch across all dimensions, and use this as the global response strength;

[0030] Based on the global response intensity, the proportion of the response intensity of each attention branch to the total response of all attention branches is calculated to obtain the modal-level relative gain distribution;

[0031] Based on the modal-level relative gain distribution, the gain distribution is used as a probability distribution to calculate the Shannon entropy, thus obtaining the modal-level relative gain entropy.

[0032] As a preferred technical solution, the process of optimizing the overall framework based on a progressive course learning mechanism also includes the following steps:

[0033] Based on training data that includes only complete modal samples, the trainable parameters in normalized flow-guided recovery and multi-expert fusion are frozen, and the multimodal feature extraction network is pre-trained by combining label smoothing cross-entropy classification loss and hard sample mining triplet loss.

[0034] Based on training data that includes only complete modality samples, the trainable parameters of normalized flow-guided recovery are activated. By randomly masking a certain modality and using the remaining available modalities as conditions, the conditional mean of the missing modality in the latent space is inferred, and recovery features are generated through inverse transformation. The trainable parameters of normalized flow-guided recovery are trained by combining spatial alignment loss, label smoothing cross-entropy classification loss, and hard sample mining triplet loss.

[0035] Based on samples including complete modal samples and multiple missing modal patterns, and combining diversity regularization loss, spatial alignment loss, label smoothing cross-entropy classification loss, and hard sample mining triplet loss, the trainable parameters in normalized flow guided recovery and multi-expert fusion, as well as the multi-modal feature extraction network, are jointly trained end-to-end.

[0036] As a preferred technical solution, the spatial alignment loss and diversity regularization loss are:

[0037]

[0038]

[0039] in, , These are spatial alignment loss and diversity regularization loss, respectively. This indicates the calculation of the expected value. It is a lightweight neural network. Indicates the missing modal features to be recovered. This represents the true missing modal features. Indicates the number of modal missing modes. This indicates the number of experts in the expert panel. This indicates the calculation of KL divergence. Indicates the first An expert on modal missing mode Expert weighting, express Dimensionally uniform distribution.

[0040] As a preferred technical solution, the vehicle image includes at least one mode of visible light image, near-infrared image, and thermal infrared image.

[0041] Another aspect of the present invention provides a multimodal vehicle re-identification system based on stream-generated multi-expert fusion, for implementing the aforementioned multimodal vehicle re-identification method, the system comprising:

[0042] The multimodal feature extraction module is used to acquire vehicle images and extract input features, classifying them into available modalities and missing modalities based on their availability.

[0043] The normalized flow-guided recovery module is used to model the joint latent distribution of multimodal modes based on normalized flow. It infers the conditional distribution parameters of missing modes in the latent space using available modes as conditions, and generates semantically consistent missing mode features through invertible transformation.

[0044] The multi-expert fusion module is used to combine missing mode-aware routing and modal-level uncertainty weighting strategy to perform semantic enhancement and bias compensation on the recovered missing modal features to obtain the vehicle identity embedding representation;

[0045] The identification module is used to achieve vehicle re-identification based on the vehicle identity embedded representation.

[0046] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0047] (1) Effective handling of various modality missing situations: This invention models the joint latent distribution of multimodal modes based on normalized flow. Using available modalities as conditions, it infers the conditional distribution parameters of missing modalities in the latent space. Through reversible transformation, it generates semantically consistent missing modal features, thereby achieving dynamic and reversible recovery of missing modalities during the inference stage. Combining missing mode-aware routing and modality-level uncertainty weighting strategies, under the premise of ensuring stable system behavior and efficient inference, it performs semantic enhancement and bias compensation on the recovered modal features, and finally generates a robust and unified vehicle identity embedding representation, which significantly improves the re-identification performance under missing modal conditions. The flow-generative multi-expert fusion framework can effectively handle different modality missing scenarios and significantly improve the robustness of the model under modality missing conditions.

[0048] (2) Maintaining modal complementarity: This invention calculates self-attention and cross-modal attention enhancement features for the features of available modalities and the features of recovered missing modalities, calculates the modal-level relative gain entropy of each modality, matches the corresponding missing modal mode, activates the corresponding expert group, realizes missing mode perception routing, and calculates expert weights based on the modal-level relative gain entropy corresponding to the modality within the activated expert group to obtain the vehicle identity embedding representation, realizes modal-level uncertainty weighting, and can effectively explore the complementary relationship between different modalities through modal perception expert branches and complexity perception routing, avoiding the loss of modal information caused by simple splicing or attention mechanisms.

[0049] (3) Accurately assess the effectiveness of the attention mechanism in a multimodal system: This invention uses modal-level relative gain entropy as a measure of feature uncertainty. A high entropy value indicates that the contributions of each attention branch are uniform, with no significant dominant branch, leading to highly unstable recovered features that require strong compensation. A low entropy value indicates that a few branches dominate the response, the attention mechanism effectively focuses, and the generated features are reliable, allowing for mild compensation or direct use. Through this entropy value, the effectiveness of the attention mechanism in a multimodal system can be dynamically assessed, providing a quantitative basis for feature recovery strategies.

[0050] (4) Overcoming gradient conflict or mode collapse problem: The present invention adopts a three-stage progressive course learning mechanism. Through the strategy of starting from simple to complex, activating modules in stages, and gradually introducing heterogeneous samples, all learnable parameters are optimized in a coordinated manner to avoid gradient conflict or mode collapse problem caused by end-to-end synchronous training, and finally achieve robust and high-precision multimodal vehicle re-identification. Attached Figure Description

[0051] Figure 1 is a flowchart of the multimodal vehicle re-identification method based on stream-generated multi-expert fusion in the embodiment;

[0052] Figure 2 is an architecture diagram of multimodal vehicle re-identification in the embodiment;

[0053] Figure 3 is a visualization analysis of the feature recovery and fusion effect in the embodiment. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0055] Example 1

[0056] To address the problems existing in the prior art, this embodiment provides a multimodal vehicle re-identification method based on stream generation multi-expert fusion, which aims to improve the robustness and accuracy of vehicle re-identification by effectively handling modality loss.

[0057] Referring to Figures 1 and 2, the method includes the following steps:

[0058] S1. Use a feature extractor to extract the original features of each modality and explicitly distinguish between available modalities and missing modalities.

[0059] During the training and inference phases, the system receives trimodal vehicle image input: visible light images. Near-infrared images Thermal infrared images Each image measures 256×128×3 pixels.

[0060] Each modality image is independently input to its corresponding feature extractor. In this embodiment, ResNet50 is used as the basic backbone network of the feature extractor, but to preserve modality specificity, the parameters of the RGB, NIR, and TIR branches are independent and do not share weights. After feature extraction, a 512-dimensional feature vector is output through a Global Average Pooling (GAP) layer.

[0061]

[0062]

[0063]

[0064] Let the total mode set be... In each forward propagation, the system receives a binary mask vector. ,in, Indicates the first One modality is available. Indicates the first One modality is missing.

[0065] Based on this division, Represents the set of available modal features. This represents the set of missing modal features.

[0066] During the training phase, the mask is randomly generated, covering all... A non-completely missing mode is selected, while a completely missing mode is meaningless and therefore excluded, in order to simulate arbitrary modal missing scenarios. During the inference phase, the mask is generated in real time by the sensor state detection module or the image quality assessment module, such as signal-to-noise ratio, contrast, and brightness threshold judgment.

[0067] The output of this step is the set of available modal features. , and a list of missing modality identifiers (mask).

[0068] S2. The normalized flow guidance module uses invertible transformation to model the joint distribution of multiple modalities, and uses available modalities as conditions to infer and recover the features of missing modalities and constrain semantic space alignment.

[0069] In this step, the basic distribution is transformed using an asymptotically invertible transformation. The data is transformed into a complex data distribution. For missing modes, the joint distribution of available modes is used as a latent variable for feature recovery. Spatial alignment loss is used to ensure the consistency between the latent space and the feature space.

[0070] S2.1 Conditional Normalized Flow Structure Design.

[0071] Normalized flow modeling is used to model the joint latent distribution of multiple modalities. This enables the inference of the conditional distribution parameters of missing modalities in the latent space, namely the mean and variance, based on the available modalities. Then, based on the expected value (mean) of the conditional distribution, semantically consistent missing modal features are generated through reversible transformation, thereby achieving dynamic and reversible recovery of missing modalities during the inference stage.

[0072] This embodiment employs a Real-valued Non-Volume Preserving (RealNVP) architecture, constructing 8 coupling layers. Let the joint feature vector of the complete modality be:

[0073]

[0074] Each of them Normalized flow model Learn about an invertible mapping:

[0075]

[0076] Make That is, the joint latent variables follow a standard Gaussian distribution, where It is an identity matrix.

[0077] During the inference phase, given the available set of modal features Its concatenated vector is denoted as The set of missing modal features is denoted as , i.e., the features to be recovered. The goal is to calculate:

[0078] or equivalent

[0079] Since the normalized flow is a reversible bijection, firstly... Mapping to the latent space via the forward transformation of the stream: (Only the dimensions of the corresponding available modalities are retained). and They are coupled to each other in the potential space.

[0080] To effectively model this conditional distribution, a conditional affine coupling mechanism is introduced at each layer of the flow. In each layer of RealNVP coupling transformation, the scaling function... With translation function The parameters are all generated by a conditional network, whose input is the available modal features. ,Right now:

[0081]

[0082] Therefore, the entire stream transformation Therefore A function with conditions is denoted as .

[0083] S2.2 Infer the conditional distribution of latent variables for missing modalities.

[0084] In the latent space, assume a joint distribution. To implicitly model a Gaussian distribution using a flow model. Given The corresponding latent variables It can be regarded as an observation, which can be obtained through forward transformation, then the missing part The conditional distribution is:

[0085]

[0086] in, Let be the covariance matrix.

[0087] In this embodiment, to simplify calculations and improve stability, the covariance matrix is ​​not explicitly calculated. Instead, the conditional mean is directly predicted using a flow model structure and a conditional network.

[0088]

[0089] in It is a lightweight neural network that can be implemented using a two-layer MLP, with its input being... The output dimension equals The dimension, i.e., the number of missing modes × 512, parameters With flow model Joint training.

[0090] This design uses available modal features as conditional input to infer the conditional distribution of missing modalities in the latent space. The inference is manifested as the prediction of the conditional mean. , is the core statistic of conditional distribution.

[0091] S2.3, Inverse transformation recovers missing modal features.

[0092] Obtain the conditional mean Then, combine it with By concatenating the vectors, a complete latent vector is formed:

[0093]

[0094] Then, through the inverse transformation of the flow model ,Will Mapping back to the original feature space yields the recovered missing modality features:

[0095]

[0096] Where [miss] represents the dimension of the missing mode in the inverse transform output.

[0097] S2.4 Spatial alignment loss function, which is used during the training phase.

[0098] To constrain recovery features Compared with true missing modal features To ensure consistency in the semantic space, a spatial alignment loss is introduced:

[0099]

[0100] in This is a lightweight projection network that can employ a single-layer MLP to align features to a shared semantic space. The loss is used only during the training phase and is computed only if the corresponding modality actually exists in the training samples, i.e., non-missing samples.

[0101] S2.5, Output.

[0102] The output of this module is the recovered feature vector of the missing modes. , with original available features By concatenating the features, a complete modal feature representation is formed:

[0103]

[0104] For use by subsequent multi-expert fusion modules.

[0105] S3. The multi-expert fusion module dynamically activates expert branches based on modality combination and feature uncertainty to achieve semantic correction and discriminative enhancement.

[0106] This step aims to construct a multi-expert dynamic fusion mechanism for practical deployment scenarios. By using hard routing for missing modes and soft weighting strategy for modal uncertainty, the mechanism performs semantic enhancement and bias compensation on the modal features recovered in step S2 while ensuring stable system behavior and efficient inference. Finally, it generates a robust and unified vehicle identity embedding representation, which significantly improves the re-identification performance under missing modal conditions.

[0107] S3.1 Multi-attention branch feature extraction.

[0108] For each available mode Calculate its self-attention and cross-modal attention enhancement features respectively, where Includes the original mode and the missing mode recovered in step S2:

[0109] One self-attention branch that captures intramodal structure:

[0110]

[0111] -1 cross-modal attention branch that introduces complementary information between modalities:

[0112]

[0113] In the formula .

[0114] in, For modality Input features ( ); It is a linear projection matrix; finally, we obtain Attention enhancement features Each dimension is .

[0115] S3.2 Calculation of relative gain entropy at the modal level.

[0116] To quantify the information enhancement effectiveness of each attention branch from a modal-wide perspective, this embodiment proposes using modal-level relative gain entropy as a measure of feature uncertainty. The calculation process is as follows:

[0117] (1) Calculate the global response intensity of each attention branch.

[0118] For modes The Attention branches ( ), calculate the sum of its exponential responses across all dimensions:

[0119]

[0120] in This indicates that the branch is in the first... The output values ​​of the dimension yield the response intensity vector:

[0121] (2) Calculate the relative gain distribution of the modal level.

[0122] For each attention branch Calculate the proportion of its response intensity to the total response of all attention branches:

[0123]

[0124] Forming a gain distribution vector: ,satisfy .

[0125] (3) Calculate the relative gain entropy of the modal level.

[0126] Treating the gain distribution as a probability distribution, calculate its Shannon entropy:

[0127]

[0128] The physical meaning of entropy is that it is close to... High entropy values ​​indicate that the contributions of each attention branch are uniform, with no significantly dominant branch, leading to highly unstable recovered features and requiring strong compensation. Low entropy values ​​close to 0 indicate that a few branches dominate the response, such as self-attention or specific cross-modal attention, indicating that the attention mechanism effectively focuses, generates reliable features, and can be lightly compensated or used directly. This entropy value allows for dynamic evaluation of the effectiveness of attention mechanisms in multimodal systems and provides a quantitative basis for feature recovery strategies.

[0129] S3.3, Two-stage expert dynamic routing with missing pattern awareness.

[0130] (1) Define 7 modes of missing modalities.

[0131] For a three-modal system consisting of RGB, NIR, and TIR, seven common missing modes are predefined. As shown in Table 1.

[0132] Table 1 Modal Missing Modes

[0133]

[0134] (2) Construct an expert pool and pattern embedding.

[0135] Building an expert pool Each mode is assigned One dedicated expert (14 experts in total);

[0136] Constructing a learnable pattern embedding matrix , used for pattern semantic encoding.

[0137] (3) Perform two-stage routing.

[0138] Phase 1: Model-level hard road.

[0139] Determine the missing mode based on the available modal combinations of the current input. Directly activate the corresponding expert group The remaining experts were blocked. The first phase features zero computation and system stability.

[0140] Phase Two: Modal-level Soft Routing.

[0141] In active Within a group of experts, modal Based on its modal level relative gain entropy Calculate expert weights:

[0142]

[0143] in, For pattern Dedicated routing weight matrix; As a linear encoding layer, it maps scalar entropy to semantic vectors. It implements intra-group adaptive compensation.

[0144] The technical effectiveness of this routing mechanism is reflected in three aspects. First, the hard routing strategy ensures system stability, enabling the same missing patterns to activate a fixed group of experts whose behavior is consistent and predictable. Second, in terms of computational efficiency, there is no computational overhead in the pattern routing stage, while soft routing only needs to be performed within two groups of experts, effectively reducing the computational load. Finally, by achieving dynamic resource allocation through soft routing based on entropy values, the utilization efficiency of computational resources can be improved while maintaining the accuracy of feature recovery.

[0145] S3.4 Compensation Feature Generation and Unified Embedding Fusion.

[0146] For each mode Its compensation enhancement features are:

[0147]

[0148] The expert input options are: splice all attention branches / weighted average / maximum response branch. In this embodiment, splicing all attention branches is selected.

[0149] Finally, the average of all available modalities is taken to generate a unified vehicle embedding:

[0150]

[0151] The embedding Used for subsequent vehicle retrieval, such as cosine similarity ranking.

[0152] S3.5, Routing diversity regularization for the training phase.

[0153] To prevent expert degradation within the same model, KL divergence loss is added during training:

[0154]

[0155] in for The loss is uniformly distributed. This loss is jointly optimized with the re-identification loss, encouraging expert division of labor.

[0156] S4. It adopts end-to-end joint training to synchronously optimize the entire system and supports adaptive recovery and deployment during inference.

[0157] To ensure the training stability and convergence efficiency of the model under complex missing modal conditions, this embodiment proposes a three-stage progressive course learning mechanism. By adopting a strategy of starting from simple to complex, activating modules in stages, and gradually introducing heterogeneous samples, all learnable parameters in steps S1 to S3 are optimized in a coordinated manner. This avoids gradient conflicts or mode collapse problems caused by end-to-end synchronous training, and ultimately achieves robust and high-precision multimodal vehicle re-identification.

[0158] S4.1, First Stage: Feature Extractor Warm-up Training

[0159] This stage uses only complete modal samples, i.e., training data available in RGB, NIR, and TIR modes, to initialize and pre-train the multimodal feature extraction network in step S1. Simultaneously, the normalized flow-guided recovery module in step S2 and the multi-expert fusion module in step S3 are frozen. During training, only the vehicle re-identification loss is optimized, which is composed of the label smoothed cross-entropy classification loss. Triple loss in hard sample mining The total loss function is expressed as:

[0160]

[0161] This loss is used to establish a preliminary discriminative feature space, avoiding noise interference introduced in the early stages due to the lack of training of the recovery module, thereby laying a stable feature representation foundation for subsequent stages.

[0162] S4.2, Second Phase: Resumption of Module Specialization Training.

[0163] This stage continues to use complete modality samples, but activates the normalized flow-guided recovery module from step S2. By randomly masking a modality and using the remaining available modalities as conditions, it infers the conditional mean of the missing modality in the latent space and generates recovered features through inverse transformation. The training loss is based on the re-identification loss and introduces the spatial alignment loss from S2.4. The total loss function is expressed as:

[0164]

[0165] in, The learnable weight parameters are initialized to 0.5 and constrained to the interval [missing value]. This is used to adaptively adjust the contribution of spatial alignment loss to the total loss, thereby guiding the recovery module to learn semantically consistent and reasonably distributed missing modal representations.

[0166] S4.3, Third Stage: Joint Fine-tuning and Expert Activation.

[0167] This stage uses mixed samples for end-to-end joint training, including complete modality samples and seven missing modality samples. Complete modality samples account for 30%, and missing modality samples account for 70%, with the seven modalities sampled evenly. Among the missing modality samples, single-missing and double-missing cases each account for 50%. This stage activates all modules, including feature extraction, missing modality recovery, multi-expert fusion, and routing mechanisms, and optimizes the complete joint loss function. This joint loss function consists of re-identification loss, spatial alignment loss, and routing diversity regularization loss. Composition, in which KL divergence was used to measure the distribution of expert route weights and uniform distribution under various missing patterns. The difference, expressed by the total loss function, is:

[0168]

[0169] in Another learnable weight parameter is initialized to 0.5 and constrained to the interval [missing value]. The strength of the adaptive balancing routing regularization term is used. Through this three-stage progressive training strategy, the model maintains training stability while gradually improving its ability to recover missing modalities and its fusion robustness, ultimately achieving high-precision, low-fluctuation multispectral vehicle re-identification performance.

[0170] S5. Experimental verification and ablation analysis.

[0171] To objectively evaluate the technical effectiveness of this method, experiments were conducted on the MSVR310 dataset. This dataset contains RGB, NIR, and TIR trimodal images, covering real-world challenging scenarios such as sensor failure, occlusion, and extreme lighting, and is an authoritative benchmark for evaluating multimodal robustness.

[0172] This embodiment uses ResNet-50 as the backbone network for the modality extractor. To ensure a fair comparison with existing work, the performance data of the comparison method CCNet is directly extracted from its original paper, "Cross-Directional Consistency Network with Adaptive Layer Normalization for Multi-Spectral Vehicle Re-Identification and a High-Quality Benchmark" (Information Fusion, 100:101901). The baseline refers to the results obtained by directly training the ResNet-50 network using real datasets. Because some earlier methods did not report results on MSVR310 or did not release their code, this embodiment only compares with CCNet, which is explicitly listed in the paper, to ensure experimental reproducibility and fairness.

[0173] The main results comparison is shown in Table 2, which demonstrates the performance of our method and CCNet on MSVR310 under different modal missing protocols. The metrics are mAP / Rank-1, in units of mAP / Rank-1.

[0174] Table 2 Performance Comparison

[0175]

[0176] As shown in Table 2, our method significantly outperforms CCNet in all missing modes, especially in bimodal missing modes, such as missing RGB and NIR, where the mAP is improved by up to 30.8 percentage points, demonstrating its strong modality recovery and fusion capabilities.

[0177] To verify the contribution of each module in this method, an ablation study was conducted on key components on MSVR310. The results are shown in Table 3, which only shows the missing RGB mode, while the missing trends of other modes are consistent.

[0178] Table 3 Ablation Results

[0179]

[0180] Among them, "+Normalized Flow Guidance Module" indicates the introduction of the normalized flow guidance module to restore the missing modality; "+Normalized Flow Guidance Module+Multi-Expert Fusion Module" is the complete scheme of this method; the fusion method of each ablation module for multimodal features in Table 2 is consistent with the splicing described in S3.4.

[0181] Ablation results show that using the normalized flow guidance module alone can improve mAP from 22.1% to 37.0% (+14.9%), verifying the effectiveness of the missing modality feature recovery mechanism of the normalized flow guidance module; using the multi-expert fusion module alone only brings limited improvement (+4.4% mAP), indicating that fusion without recovery cannot make up for missing information; the combined use of the normalized flow guidance module and the multi-expert fusion module brings synergistic gains (+27.4% mAP vs. ResNet50), proving that the two together constitute the core innovation of this invention.

[0182] To further qualitatively verify the effectiveness of each module of this invention, a visualization analysis of the feature recovery and fusion process on the MSVR310 dataset was performed, and the results are shown in Figure 3. (a) shows the residual curves between the missing modal features recovered from the other two available modalities and the corresponding true features in the case of single-modal missing features (RGB, NIR, and TIR missing, respectively), guided by the normalized flow module. The horizontal axis represents the feature dimension, the vertical axis represents the average L1 residual value in each dimension, and the error band represents the standard deviation. (b) shows the visualization results of the t-SNE feature embedding in the case of dual-modal missing features (RGB and NIR missing, with only TIR available). Base represents the feature distribution under complete trimodal input, Pre represents the feature distribution after missing modal recovery but before processing by the multi-expert fusion module, and Post represents the feature distribution after processing by the multi-expert fusion module of this method.

[0183] As shown in Figure 3(a), in the three single-modal missing cases of "missing RGB", "missing NIR", and "missing TIR", the residual difference between the missing modal features recovered from the other two modalities by the normalized flow guidance module and the true features is small, indicating that the recovered features are highly aligned with the true features in terms of distribution. This result verifies that the normalized flow guidance module can effectively maintain the distribution consistency of cross-modal features through reversible flow modeling, thus providing reliable input for subsequent fusion.

[0184] As shown in Figure 3(b), in the extreme case of "missing RGB and NIR," i.e., only TIR is available, the feature Pre without multi-expert fusion module processing shows a significant offset from the complete modal feature Base. However, after processing by the multi-expert fusion module Post of our method, the feature distribution significantly converges to the vicinity of the complete modality Base, and the bias region is effectively suppressed. This phenomenon indicates that the multi-expert fusion module can generate compensatory features through uncertain dynamic routing, effectively mitigating the representation bias caused by modality loss.

[0185] The above visualization results corroborate the quantitative ablation experiments in Table 2, jointly supporting the technical effectiveness of the two core technical features of normalized flow-guided recovery and multi-expert dynamic correction in this embodiment.

[0186] Example 2

[0187] Building upon Example 1, this example provides a multimodal vehicle re-identification system based on stream-generated multi-expert fusion to implement the multimodal vehicle re-identification method of Example 1. Referring to Figure 2, the system input is a three-channel 256×256 pixel image. The existing recognition module is used to obtain the recognition result from the output, a 512-dimensional vehicle identity embedding vector, achieving cross-camera vehicle matching. The overall network architecture includes the following core components:

[0188] (1) Multimodal feature extraction module, used to acquire vehicle images, independently encode the input images of RGB, NIR, TIR and other modalities to obtain input features, and classify them into available modalities and missing modalities according to their availability.

[0189] (2) The normalized flow-guided recovery module is used to model the joint latent distribution of multimodal modes based on normalized flow. Using available modes as conditions, it infers the conditional distribution parameters of missing modes in the latent space and generates semantically consistent missing mode features through invertible transformation.

[0190] (3) A multi-expert fusion module is used to combine missing mode perception routing and modal-level uncertainty weighting strategy to perform semantic enhancement and bias compensation on the recovered missing modal features to obtain the vehicle identity embedding representation. The module includes multiple modal combination perception expert branches. Based on the input modal combination and recovery uncertainty, the corresponding expert branches in the missing modal state are activated to optimize computational efficiency and energy consumption. The attention branches are dynamically activated through modal-level relative gain entropy to process data, thereby fusing and outputting a highly discriminative embedding vector.

[0191] (4) Identification module, used to realize vehicle re-identification based on the vehicle identity embedding representation.

[0192] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal vehicle re-identification method based on stream-generative multi-expert fusion, characterized in that, The process includes the following steps: acquiring vehicle images and extracting input features, classifying modalities into available modalities and missing modalities based on their availability; modeling the joint latent distribution of multimodalities based on normalized flow, inferring the conditional distribution parameters of missing modalities in the latent space using available modalities as conditions, and generating semantically consistent missing modal features through invertible transformation. Combining missing mode-aware routing and modal-level uncertainty weighting strategies, semantic enhancement and bias compensation are performed on the recovered missing modal features to obtain a vehicle identity embedding representation; vehicle re-identification is achieved based on the vehicle identity embedding representation. The process of obtaining the vehicle identity embedding representation includes the following steps: calculating self-attention and cross-modal attention enhancement features for the features of available modalities and the features of recovered missing modalities, respectively. Based on the self-attention and cross-modal attention enhancement features, the modal-level relative gain entropy of each modality is calculated; based on the available modalities and missing modalities, the corresponding missing modal patterns are matched, and the corresponding expert groups are activated to realize missing mode-aware routing; within the activated expert groups, the expert weights are calculated based on the modal-level relative gain entropy corresponding to the modality. Based on the expert weights, compensated enhancement features are generated to obtain the vehicle identity embedding representation, realizing modal-level uncertainty weighting. The calculation process of the modal-level relative gain entropy includes the following steps: calculating the sum of the exponential responses of each attention branch in all dimensions as the global response intensity; Based on the global response intensity, the proportion of the response intensity of each attention branch to the total response of all attention branches is calculated to obtain the modal-level relative gain distribution; based on the modal-level relative gain distribution, the gain distribution is used as a probability distribution to calculate the Shannon entropy to obtain the modal-level relative gain entropy.

2. The multimodal vehicle re-identification method based on stream-generated multi-expert fusion according to claim 1, characterized in that, The extraction of input features is implemented based on a multimodal feature extraction network, including: extracting features according to the modality type of the vehicle image using the corresponding feature extractor, and performing global average pooling to obtain the input features.

3. The multimodal vehicle re-identification method based on stream-generated multi-expert fusion according to claim 1, characterized in that, The process of generating semantically consistent missing modal features through reversible transformation includes the following steps: using a flow model based on multi-layer conditional affine coupling, the concatenated vector of available modalities is forward transformed and mapped to the latent space. In each layer of the flow, the parameters of the scaling function and translation function are generated by a conditional network, the input of which is the concatenated vector of available modalities; the concatenated vector of available modalities is used as the input of a lightweight neural network, and the conditional mean of the missing modal features is output; the conditional mean is concatenated with the concatenated vector of available modalities to obtain a latent vector, which is then mapped back to the original feature space through the inverse transformation of the flow model to obtain the recovered missing modal features.

4. The multimodal vehicle re-identification method based on stream-generated multi-expert fusion according to claim 3, characterized in that, The lightweight neural network is trained in conjunction with the flow model.

5. The multimodal vehicle re-identification method based on stream-generated multi-expert fusion according to claim 1, characterized in that, It also includes a process of optimizing the overall framework based on a progressive course learning mechanism, including the following steps: Based on training data containing only complete modality samples, freeze the trainable parameters in normalized flow-guided recovery and multi-expert fusion, and pre-train the multimodal feature extraction network by combining label smoothing cross-entropy classification loss and hard sample mining triplet loss; Based on training data containing only complete modality samples, activate the trainable parameters of normalized flow-guided recovery, infer the conditional mean of the missing modality in the latent space by randomly masking a certain modality and using the other available modalities as conditions, and generate recovery features through inverse transformation, and train the trainable parameters of normalized flow-guided recovery by combining spatial alignment loss, label smoothing cross-entropy classification loss and hard sample mining triplet loss; Based on samples including complete modality samples and multiple missing modality patterns, combine diversity regularization loss, spatial alignment loss, label smoothing cross-entropy classification loss and hard sample mining triplet loss to perform end-to-end joint training of the trainable parameters in normalized flow-guided recovery and multi-expert fusion and the multimodal feature extraction network.

6. The multimodal vehicle re-identification method based on stream-generated multi-expert fusion according to claim 5, characterized in that, The spatial alignment loss and diversity regularization loss are: in, 、 These are spatial alignment loss and diversity regularization loss, respectively. This indicates the calculation of the expected value. It is a lightweight neural network. Indicates the recovered missing modal features, This represents the true missing modal features. Indicates the number of modal missing modes. This indicates the number of experts in the expert panel. This indicates the calculation of KL divergence. Indicates the first An expert on modal missing mode Expert weighting, express Dimensionally uniform distribution.

7. The multimodal vehicle re-identification method based on stream-generated multi-expert fusion according to claim 1, characterized in that, The vehicle image includes at least one modality of visible light image, near-infrared image, and thermal infrared image.

8. A multimodal vehicle re-identification system based on stream-generated multi-expert fusion, characterized in that, To implement the multimodal vehicle re-identification method as described in any one of claims 1-7, the system comprises: a multimodal feature extraction module for acquiring vehicle images and extracting input features, classifying modalities into available modalities and missing modalities based on their availability; a normalized flow-guided recovery module for modeling the joint latent distribution of multimodalities based on normalized flow, inferring the conditional distribution parameters of missing modalities in the latent space using available modalities as conditions, and generating semantically consistent missing modal features through invertible transformation; a multi-expert fusion module for combining missing mode-aware routing and modality-level uncertainty weighting strategies to perform semantic enhancement and bias compensation on the recovered missing modal features to obtain a vehicle identity embedding representation; and an identification module for realizing vehicle re-identification based on the vehicle identity embedding representation.

Images