Target detection methods, devices, and electronic equipment based on multimodal dynamic fusion

Abstract

This invention relates to the field of target detection technology, and provides a target detection method, device, and electronic device based on multimodal dynamic fusion. The method includes: a collaborative alignment module performing semantic alignment on visible light modal features and infrared modal features to obtain a first feature to be fused and a second feature to be fused, thereby effectively bridging the semantic differences between different modalities and reducing the interference of modality-specific noise on the expression of shared features; an expert hybrid network in the dynamic fusion module determines the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and performs feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain fused features, so that the target detection model can adaptively adjust the degree of dependence on visible light and infrared modalities according to specific scenarios, avoiding the excessive dependence of static fusion strategies on failed modes, and improving the adaptability and reliability of the target detection model in complex and changing environments while improving the quality of feature fusion.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a target detection method, apparatus and electronic device based on multimodal dynamic fusion. Background Technology

[0002] Multimodal target detection, as an important research direction in computer vision and intelligent perception, has received widespread attention in recent years. Compared with single-modal detection, fusing information from multiple sensor sources can more comprehensively characterize target features, thereby improving the accuracy and robustness of target detection in complex environments. In particular, the combination of visible light (RGB) and infrared (IR) modes has shown unique advantages in applications such as low-altitude remote sensing, autonomous driving, security monitoring, and aerospace.

[0003] Infrared-visible light fusion technology, through its complementary fusion mechanism, can collaboratively capture more comprehensive target feature information. This effectively overcomes the inherent detection performance limitations of single-modal systems, thereby significantly improving the detection accuracy and robustness of the model. Currently, research on multimodal target detectors focuses on the innovation and optimization of fusion strategies, and various representative frameworks have been proposed in related studies.

[0004] However, existing technologies still have limitations. First, features from different modalities exhibit significant semantic differences, often leading to interference in direct alignment and fusion. For example, RGB images primarily characterize the structure and texture details of objects, while infrared images encode thermal radiation patterns. During direct fusion, modality-specific features can easily interfere with the extraction of cross-modal shared features (such as object contours and positions), resulting in modality-specific noise embedded in the shared features and affecting detection accuracy. Second, existing fusion strategies are mostly based on early fusion, late fusion, or feature stitching, typically treating different modalities as equally weighted inputs, failing to fully consider the differences in importance between different modalities. In low-light nighttime scenes, infrared modalities are more stable and reliable than RGB modalities, but static fusion methods cannot dynamically adjust modality weights, often resulting in over-reliance on invalid RGB information, leading to decreased detection performance and weakening the advantages of modal complementarity. Therefore, existing methods still lack sufficient adaptability and robustness in complex environments. Summary of the Invention

[0005] This invention provides a target detection method, device, and electronic device based on multimodal dynamic fusion, which addresses the shortcomings of existing infrared-visible light fusion technology in terms of adaptability and robustness in complex environments.

[0006] This invention provides a target detection method based on multimodal dynamic fusion, comprising the following steps: Acquire visible light and infrared images, and extract features from the visible light and infrared images respectively to obtain visible light modal features and infrared modal features; The visible light modal features and the infrared modal features are input into the target detection model to obtain the target detection result output by the target detection model; The target detection model includes a collaborative alignment module, a dynamic fusion module, and a detection layer; The collaborative alignment module is used to perform semantic alignment on the visible light modal features and the infrared modal features to obtain a first feature to be fused and a second feature to be fused. The dynamic fusion module includes an expert hybrid network, which is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain fused features; The detection layer is used to determine the target detection result based on the fusion features.

[0007] According to the target detection method based on multimodal dynamic fusion provided by the present invention, the collaborative alignment module is specifically used for: The visible light mode features and the infrared mode features are decoupled to obtain mode-shared features, visible light mode-specific features, and infrared mode-specific features; The modality sharing features are enhanced to obtain enhanced modality sharing features; The visible light mode-specific features and the infrared mode-specific features are purified to obtain pure visible light mode-specific features and pure infrared mode-specific features; Based on the enhanced modal sharing features, the pure visible light modal-specific features, and the pure infrared modal-specific features, cross-modal information compensation is performed to obtain the first feature to be fused and the second feature to be fused.

[0008] According to a target detection method based on multimodal dynamic fusion provided by the present invention, the step of decoupling the visible light modal features and the infrared modal features to obtain modality-shared features, visible light modality-specific features, and infrared modality-specific features includes: The mode-shared features are obtained by performing element-wise averaging on the visible light mode features and the infrared mode features; Perform a first element-wise difference operation on the visible light mode features and the infrared mode features to obtain the visible light mode-specific features; A second element-wise difference operation is performed on the infrared modal features and the visible light modal features to obtain the infrared modal-specific features.

[0009] According to the target detection method based on multimodal dynamic fusion provided by the present invention, the enhancement processing of the modality-shared features to obtain enhanced modality-shared features includes: The modality sharing features are input into the gating network to obtain the routing weights corresponding to multiple expert networks output by the gating network; Based on the parallel processing of the modality-shared features by the multiple expert networks, multiple candidate enhanced shared features are obtained; The enhanced modality shared features are obtained by weighted summation of the multiple candidate enhanced shared features based on the routing weights.

[0010] According to the present invention, a target detection method based on multimodal dynamic fusion is provided, wherein the dynamic fusion module includes multiple fusion modules; and the collaborative alignment module includes multiple semantic alignment modules. The step of fusing the first feature to be fused and the second feature to be fused based on the dynamic fusion weight to obtain fused features includes: Based on any of the fusion modules and the dynamic fusion weights in any of the fusion modules, feature fusion is performed on the first feature to be fused and the second feature to be fused output by any of the semantic alignment modules to obtain any multi-scale fusion feature; The fusion features are determined based on all the aforementioned multi-scale fusion features.

[0011] According to the present invention, a target detection method based on multimodal dynamic fusion is provided, wherein the fusion module is used for: Based on the scene information of the first feature to be fused and the second feature to be fused received by the fusion module, the primary modal feature and the secondary modal feature in the first feature to be fused and the second feature to be fused are determined; The submodal features are fused into the main modal features to obtain the multi-scale fused features.

[0012] According to the target detection method based on multimodal dynamic fusion provided by the present invention, the step of fusing the submodal features into the main modal features to obtain the multi-scale fused features includes: The main modal features, the intra-layer fusion features generated by the expert hybrid network, the mapped submodal features, the historical fusion results, and the static random noise features are concatenated to obtain concatenated features. The intra-layer fusion features are determined based on the first and second features to be fused received by the current fusion module. The mapped submodal features are obtained by performing dimensionality mapping on the submodal features in the current fusion module. The historical fusion results are the multi-scale fusion features output by the previous fusion module. The static random noise features are preset random numerical tensors. The splicing features are processed by the dimension transformation module to obtain the intermediate fused representation; The intermediate fusion representation is residually connected to the main modality feature to obtain the multi-scale fusion feature.

[0013] The present invention also provides a target detection device based on multimodal dynamic fusion, comprising the following units: The acquisition unit is used to acquire visible light images and infrared images, and to extract features from the visible light images and the infrared images respectively to obtain visible light mode features and infrared mode features; The input unit is used to input the visible light modal features and the infrared modal features into the target detection model to obtain the target detection result output by the target detection model; The target detection model includes a collaborative alignment module, a dynamic fusion module, and a detection layer; The collaborative alignment module is used to perform semantic alignment on the visible light modal features and the infrared modal features to obtain a first feature to be fused and a second feature to be fused. The dynamic fusion module includes an expert hybrid network, which is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain fused features; The detection layer is used to determine the target detection result based on the fusion features.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method based on multimodal dynamic fusion as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target detection method based on multimodal dynamic fusion as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the target detection method based on multimodal dynamic fusion as described above.

[0017] The target detection method, apparatus, and electronic device based on multimodal dynamic fusion provided by this invention, on the one hand, uses a collaborative alignment module to semantically align visible light modal features and infrared modal features to obtain a first feature to be fused and a second feature to be fused, thereby effectively bridging the semantic differences between different modalities and reducing the interference of modality-specific noise on the expression of shared features; on the other hand, the expert hybrid network in the dynamic fusion module is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain fused features. This enables the target detection model to adaptively adjust its dependence on visible light and infrared modalities according to specific scenarios, avoiding the excessive dependence of static fusion strategies on failed modalities. Thus, while improving the quality of feature fusion, it significantly enhances the adaptability and reliability of the target detection model in complex and variable environments, ultimately ensuring the stability and robustness of detection accuracy. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the target detection method based on multimodal dynamic fusion provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the enhanced modal sharing feature determination provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the MOE structure provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the dynamic layered fusion module provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the target detection model provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the target detection device based on multimodal dynamic fusion provided by the present invention.

[0025] Figure 7This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] In related technologies, visible light images provide rich texture, shape, and color information, making them suitable for target recognition under well-lit conditions; while infrared images, by capturing the thermal radiation characteristics of objects, can maintain stable imaging under low light, obstruction, or adverse weather conditions. The complementarity of these two technologies provides crucial support for all-weather, all-scenario intelligent perception. For example, in nighttime traffic monitoring, the RGB mode is susceptible to insufficient light, while the infrared mode can still clearly capture vehicle outlines; in complex backgrounds or camouflaged environments, the infrared mode can help suppress interference information and improve detection robustness.

[0028] XX utilizes a symmetric and compact multimodal fusion strategy to supplement information and enhances the representation of small targets through an auxiliary super-resolution branch. This auxiliary branch is discarded during inference to maintain efficient detection. DEYOLO integrates a dual semantic-spatial enhancement module and a dual-mechanism optimization. It aggregates RGB thermal modal information in the feature space while suppressing interference and expands the receptive field by combining bidirectional decoupling focusing, significantly improving the accuracy of low-light target detection. XX et al., through cross-modal channel and spatial feature difference modeling mechanisms, combined with Transformer-based cross-attention and self-attention strategies for long-range interaction, achieved the synergistic fusion of complementary information and cooperative features of visible and infrared modalities within a multi-branch structure framework, thereby achieving robust all-weather adaptive target detection in complex environments.

[0029] However, existing technologies still have limitations. First, features of different modalities exhibit significant semantic differences, often leading to interference with direct alignment and fusion. Second, existing fusion strategies are mostly based on early fusion, late fusion, or feature splicing, typically treating different modalities as equally weighted inputs, failing to fully consider the differences in importance between modalities. In low-light nighttime scenes, infrared modalities are more stable and reliable than RGB modalities, but static fusion methods cannot dynamically adjust modal weights, often resulting in over-reliance on invalid RGB information, leading to decreased detection performance and weakening the advantages of modal complementarity. Therefore, existing methods still lack sufficient adaptability and robustness in complex environments.

[0030] To address the aforementioned problems, this invention provides a target detection method based on multimodal dynamic fusion. This method can be applied to fields requiring all-weather, all-scenario target detection, such as autonomous driving, security monitoring, drone inspection, and low-altitude remote sensing. This method effectively fuses complementary information from visible light and infrared images through a novel target detection model, solving the problems of semantic misalignment caused by modal differences and poor adaptability of static fusion strategies in existing technologies. This significantly improves the accuracy and robustness of target detection in complex environments. Figure 1 This is a flowchart illustrating the target detection method based on multimodal dynamic fusion provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110 and 120.

[0031] Step 110: Acquire visible light images and infrared images, and extract features from the visible light images and infrared images respectively to obtain visible light mode features and infrared mode features.

[0032] Specifically, visible light images and infrared images are acquired. Visible light images, also known as RGB images, are captured by visible light sensors, such as CMOS or CCD cameras. Visible light images are rich in color, texture, and detail information, clearly characterizing the appearance of objects and possessing high recognizability in well-lit environments. Infrared images, also known as IR images or thermal imaging images, are formed by capturing the thermal radiation of an object using an infrared sensor. They do not depend on ambient light but rather on the object's temperature differences for imaging. Therefore, infrared images can reliably detect targets with thermal characteristics, such as pedestrians and vehicle engines, even in scenarios where visible light images are ineffective, such as low light, nighttime, or smoke.

[0033] There are several ways to acquire visible light and infrared images. In a typical application scenario, a visible light camera and an infrared camera can be integrated into a single device platform, such as a vehicle, drone, or surveillance pole. To ensure the effectiveness of subsequent fusion, these two cameras are usually pre-calibrated to achieve spatial alignment and temporal synchronization of image acquisition, thereby obtaining a pair of visible light and infrared images for the same scene. Alternatively, in other implementations, pre-acquired and paired visible light and infrared image data can be read from a database or storage device.

[0034] After acquiring visible light and infrared images, feature extraction can be performed on each image to obtain visible light modal features and infrared modal features. The purpose of this step is to transform high-dimensional pixel-level data into a more compact, semantically informative low-dimensional feature representation. Here, CNNs (Convolutional Neural Networks), such as ResNet (Residual Network), or more advanced Transformer-based network structures can be used to extract features from the visible light and infrared images respectively.

[0035] In this embodiment of the invention, a Siamese Network architecture can be employed, which uses two feature extraction networks with identical structures and weights that can be shared or not to process visible light images and infrared images respectively. After processing by the feature extraction networks, the visible light image is converted into visible light modal features, and the infrared image is converted into infrared modal features. Visible light modal features and infrared modal features are typically represented as multidimensional tensors, which contain semantic information of the image at different levels of abstraction, such as edges, contours, components, and combinations of objects.

[0036] Step 120: Input the visible light modal features and the infrared modal features into the target detection model to obtain the target detection result output by the target detection model; The target detection model includes a collaborative alignment module, a dynamic fusion module, and a detection layer; The collaborative alignment module is used to perform semantic alignment on the visible light modal features and the infrared modal features to obtain a first feature to be fused and a second feature to be fused. The dynamic fusion module includes an expert hybrid network, which is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain fused features; The detection layer is used to determine the target detection result based on the fusion features.

[0037] Specifically, after obtaining the visible light modal features and infrared modal features, these features can be input into the target detection model to obtain the target detection results output by the model. The target detection model includes a collaborative alignment module, a dynamic fusion module, and a detection layer.

[0038] Specifically, the collaborative alignment module performs semantic alignment on the input visible light modal features and infrared modal features, resulting in a first feature to be fused and a second feature to be fused. Semantic alignment aims to address the modal differences between the two types of features. Because the imaging mechanisms of visible light and infrared images are completely different, although their extracted features may point to the same physical object, their representations in the feature space differ significantly. For example, visible light modal features focus more on texture, while infrared modal features focus more on thermal distribution. Directly fusing these two semantically different features may lead to information conflicts and noise interference, thus reducing detection performance. Therefore, the collaborative alignment module adjusts the features of the two modalities through specific transformations or interaction mechanisms, making their feature representations more consistent and compatible at the semantic level, laying the foundation for subsequent efficient fusion.

[0039] The first and second features to be fused here are the visible light modal features and infrared modal features after semantic alignment. They are derived from the original visible light modal features and infrared modal features, respectively, but their representation forms have become more coordinated and consistent.

[0040] The dynamic fusion module includes a Mixture of Experts (MoE) network, which is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain the fused feature.

[0041] The expert hybrid network is a neural network model consisting of multiple expert sub-networks and a gating network. The gating network dynamically assigns weights to each expert based on the characteristics of the input data, and then weights and combines the outputs of all experts. In this embodiment, the expert hybrid network analyzes the scene information contained in the first and second features to be fused, such as determining whether the current scene is daytime or nighttime, and accordingly determines a dynamic fusion weight. The dynamic fusion weight is not fixed but adaptively adjusted based on the content of each pair of input images. For example, in a well-lit daytime scene, the visible light image quality is high, and the expert hybrid network may assign a larger weight to the first feature to be fused (originating from visible light); while in nighttime or inclement weather scenes, infrared images are more reliable, and the second feature to be fused (originating from infrared) may be assigned an even larger weight. Subsequently, the dynamic fusion module performs feature fusion on the first and second features based on the dynamic fusion weight.

[0042] Here, feature fusion is performed on the first feature to be fused and the second feature to be fused based on dynamic fusion weights. This can be done by concatenating the first feature to be fused and the second feature to be fused based on dynamic fusion weights, or by using an attention mechanism to weight the first feature to be fused and the second feature to be fused before concatenation. This embodiment of the invention does not specifically limit this.

[0043] Finally, the detection layer is the output of the object detection model. It receives the fused features generated by the preceding modules as input. Because the fused features have already combined the advantages of both modalities and eliminated most conflicts, they contain very high-quality semantic representations. The detection layer can be a detection head common in YOLO series models. Based on the high-quality fused features, the detection layer can perform classification and regression predictions to ultimately determine the object detection result.

[0044] Here, the object detection results typically include the category of each detected object in the scene, such as people, vehicles, location information, and the corresponding confidence score. This embodiment of the invention does not impose specific limitations on this.

[0045] The method provided in this invention, on the one hand, uses a collaborative alignment module to semantically align visible light modal features and infrared modal features to obtain a first feature to be fused and a second feature to be fused, thereby effectively bridging the semantic differences between different modalities and reducing the interference of modality-specific noise on the expression of shared features; on the other hand, the expert hybrid network in the dynamic fusion module is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain fused features. This enables the target detection model to adaptively adjust its dependence on visible light and infrared modalities according to specific scenarios, avoiding the excessive dependence of static fusion strategies on failed modalities. Thus, while improving the quality of feature fusion, it significantly enhances the adaptability and reliability of the target detection model in complex and variable environments, ultimately ensuring the stability and robustness of detection accuracy.

[0046] Based on the above embodiments, the collaborative alignment module is specifically used for: Step 210: Decouple the visible light mode features and the infrared mode features to obtain mode-shared features, visible light mode-specific features, and infrared mode-specific features; Step 220: Enhance the modal sharing features to obtain enhanced modal sharing features; Step 230: Purify the visible light mode-specific features and the infrared mode-specific features to obtain pure visible light mode-specific features and pure infrared mode-specific features; Step 240: Based on the enhanced modal sharing features, the pure visible light modal-specific features, and the pure infrared modal-specific features, cross-modal information compensation is performed to obtain the first feature to be fused and the second feature to be fused.

[0047] Specifically, the workflow of the collaborative alignment module is as follows: First, the visible light modal features and infrared modal features are decoupled to obtain modality-shared features, visible light modality-specific features, and infrared modality-specific features. Here, decoupling refers to separating the information with different properties contained in the two types of input modal features.

[0048] Here, modal shared features refer to common information that exists stably in both visible and infrared modes, such as the basic outline and structure of an object—general representations independent of the imaging mode. Visible light modal-specific features refer to information that is significant only in the visible light mode, such as the color and surface texture of an object. Infrared modal-specific features refer to information that is significant only in the infrared mode, such as the thermal radiation distribution and temperature differences of an object. Through decoupling operations, the original, mixed modal features can be decomposed into three purer, more specific components, laying the foundation for subsequent targeted processing and compensation.

[0049] Secondly, the modality-shared features are enhanced to obtain enhanced modality-shared features. Since modality-shared features represent the consensus between two modalities and are fundamental to object detection tasks, enhancing them is crucial. The enhancement process aims to improve the representational power and robustness of the shared features, for example, by suppressing noise and strengthening key semantic information, making these common features more prominent and reliable.

[0050] Next, the visible light mode-specific features and infrared mode-specific features are purified to obtain pure visible light mode-specific features and pure infrared mode-specific features. After initial decoupling, some shared information or noise from another mode may still remain in the mode-specific features. The purpose of the purification process is to further remove these residual mixed information, so that the visible light mode-specific features and infrared mode-specific features each contain only the most unique and purest information of their respective modes.

[0051] in, This indicates the unique characteristics of the pure infrared mode. This indicates the unique characteristics of pure visible light modes. Indicates infrared modal characteristics, Represents the characteristics of visible light modes. This indicates a max pooling operation. This indicates the average pooling operation. The operation involves first performing max pooling on the features, then average pooling on the features, and finally fusing the features obtained from the max pooling and average pooling operations to obtain the fused features, which are then activated using an activation function. Represents element-wise addition. This represents element-wise multiplication. () represents the activation function. Indicates the unique characteristics of infrared modes, This indicates the unique characteristics of visible light modes.

[0052] Here, the initial differential features are pooled using MP-Pool to generate descriptor vectors representing the importance of channels. These vectors are based on the fusion of global statistical characteristics and local salient responses, accurately quantifying the contribution of each channel to modality-specific information. Then, this attention weight is multiplied with the original modality input features channel by channel to achieve the purpose of adaptively filtering and enhancing core modality-specific information channels, ultimately extracting purer modality-specific features with stronger discriminative power.

[0053] For example, any weak temperature-related signals are removed from the visible light mode-specific features, and any color-related interference is removed from the infrared mode-specific features. After purification, the resulting pure visible light mode-specific features and pure infrared mode-specific features can more accurately provide supplementary information that the other mode lacks.

[0054] Finally, based on the enhanced modal shared features, the pure visible light modal-specific features, and the pure infrared modal-specific features, cross-modal information compensation is performed to obtain the first and second features to be fused. In this step, the enhanced modal shared features, the pure visible light modal-specific features, and the pure infrared modal-specific features, after the above processing, can be organically recombined. One possible implementation is to fuse the enhanced shared features with the two pure modal-specific features respectively, thereby using the unique information of one modality to supplement the other. For example, pure infrared modal-specific features, such as hotspot information, can be added to the visible light modal feature stream to generate the first feature to be fused. This allows even insignificant targets in the visible light image, such as camouflaged people, to be enhanced due to their thermal signals. Similarly, pure visible light modal-specific features, such as texture details, can be added to the infrared modal feature stream to generate the second feature to be fused. The resulting first and second features to be fused both absorb the advantageous information of the other modality, making them more semantically aligned and complementary.

[0055] The formulas for the first feature to be fused and the second feature to be fused are as follows: in, Indicates the first feature to be fused. Indicates the second feature to be fused. Represents the characteristics of visible light modes. Indicates infrared modal characteristics, This represents the first component in the enhanced modality-shared features. This represents the second component in the enhanced modality-sharing features. This indicates the unique characteristics of pure visible light modes. This indicates the unique characteristics of the pure infrared mode. These are learnable, adaptive weight coefficients used to optimize the performance of object detection models.

[0056] Understandably, this involves introducing refined, unique information from complementary modalities to facilitate alignment of unique semantic spaces; and enhancing modality-shared features to strengthen the shared semantic base within each modality.

[0057] The method provided in this invention, through the above four-step processing of decoupling, enhancement, purification and compensation, enables the collaborative alignment module to systematically separate, optimize and reorganize multimodal information, effectively solving the semantic gap problem between modalities, and providing input features with higher information quality and better alignment effect for the subsequent dynamic fusion module, thereby improving the accuracy and reliability of target detection.

[0058] Based on the above embodiments, step 210 includes: Step 211: Perform element-wise averaging on the visible light modal features and the infrared modal features to obtain the modal shared features; Step 212: Perform a first element-wise difference operation on the visible light mode features and the infrared mode features to obtain the visible light mode-specific features; Step 213: Perform a second element-wise difference operation on the infrared modal features and the visible light modal features to obtain the infrared modal-specific features.

[0059] Specifically, firstly, element-wise averaging can be performed on the visible light modal features and infrared modal features to obtain modal-shared features. Then, a first element-wise difference operation is performed on the visible light modal features and infrared modal features to obtain visible light modal-specific features, and a second element-wise difference operation is performed on the infrared modal features and visible light modal features to obtain infrared modal-specific features.

[0060] In one embodiment, the unique features of IR and RGB modes are initially defined using a difference method. Given that each modal feature can be decomposed into modality-specific features and modality-shared features, i.e. ,in, Indicates modality-specific characteristics, This represents modal shared features, from which the expression for modal shared features can be derived. The principle behind this operation is that if a feature exists in both modalities, their values ​​are superimposed and averaged, thus being preserved. If a feature exists only in one modality, the averaging operation halves its strength, relatively weakening its status as a shared feature. Therefore, the averaged result can well approximate the common information of the two modalities, i.e., modal shared features.

[0061] The principle of the first element-wise difference operation is that by subtracting the infrared mode features from the visible light mode features, the shared parts of the two modes will cancel each other out, while features unique to the visible light mode that are weaker or absent in the infrared mode (such as color and texture) will be highlighted. Therefore, the difference result can effectively extract the features unique to the visible light mode.

[0062] The second element-wise difference operation subtracts the visible light mode features from the infrared mode features, canceling out the shared parts, thereby highlighting the unique features in the infrared mode that are weak or absent in the visible light mode (such as thermal radiation differences), and effectively extracting the unique features of the infrared mode.

[0063] The method provided in this invention offers an extremely simple and efficient feature decoupling scheme. This scheme eliminates the need for complex network structures and extensive parameter learning; it achieves effective separation of modality-shared features and modality-specific features through basic mathematical operations alone. This significantly reduces the computational complexity of the target detection model and improves processing efficiency.

[0064] Based on the above embodiments, step 220 includes: Step 221: Input the modality sharing features into the gating network to obtain the routing weights corresponding to multiple expert networks output by the gating network; Step 222: Based on the multiple expert networks, the modality-shared features are processed in parallel to obtain multiple candidate enhanced shared features; Step 223: Based on the routing weights, perform a weighted summation of the multiple candidate enhanced shared features to obtain the enhanced modality shared features.

[0065] Specifically, the essential difference between multimodalities lies in the modality-specific features inherent in the semantic representation spaces of different modalities. The differences lie in their physical perception mechanisms: for example, IR represents the thermal radiation temperature field, while RGB represents the reflectance spectral texture color. However, since both observe the same physical entity, their semantic spaces necessarily share a semantic basis, i.e., modal shared features. Directly forcing the fusion of IR and RGB features will result in issues due to their modal-specific characteristics. The heterogeneity leads to modal conflict.

[0066] Accordingly, firstly, the modality-shared features can be input into a gating network to obtain the routing weights corresponding to multiple expert networks output by the gating network. The gating network is typically a small neural network that receives the modality-shared features as input, analyzes the content of those features, and outputs a set of routing weights, the number of which is the same as the number of subsequent expert networks. Each weight value indicates the extent to which the input feature should be processed by the corresponding expert network.

[0067] Furthermore, modality-shared features can be processed in parallel using multiple expert networks to obtain multiple candidate enhancement features. Here, multiple expert networks are multiple parallel sub-networks with similar or different structures. Each expert network is adapted to a specific feature transformation, such as edge sharpening, noise suppression, or texture enhancement. The shared features of each modality are input in parallel to all expert networks, and after processing by each network, a diverse set of candidate enhancement features is generated.

[0068] Finally, the enhanced modality shared features are obtained by weighted summation of multiple candidate enhanced shared features based on routing weights.

[0069] In one embodiment, this step aims to refine modality-shared features. It also enhances the common feature components within each mode. The MoE mechanism is used to improve... The representational capability of MoE. MoE contains N expert networks with identical structures. Each expert is defined as Gated networks According to the input Dynamically calculate the weights of each expert The formula is as follows: in, It is a learnable adaptive scaling weight, which MoE focuses on through ensemble learning. From different aspects, obtain more accurate enhanced modal sharing features. , This indicates modality sharing features.

[0070] Figure 2 This is a schematic diagram of the enhanced modal sharing feature determination provided by the present invention, as shown below. Figure 2 As shown, using precise Together with information from another modality, attention weights are generated through MP-Pool operations, and common feature components are extracted from and enhanced from the original features of this modality. in, This represents the first component in the enhanced modality-shared features. This represents the second component in the enhanced modality-sharing features. This indicates enhanced modality sharing features. Represents the characteristics of visible light modes. Indicates infrared modal characteristics.

[0071] The method provided in this invention employs an enhancement mechanism based on gating and expert networks, enabling the model to flexibly select and combine the most suitable enhancement strategies according to the dynamic changes in input features, rather than using a fixed, unchanging enhancement method. This adaptive enhancement processing can more finely improve the quality of modality-shared features.

[0072] Based on the above embodiments, the dynamic fusion module includes multiple fusion modules; the collaborative alignment module includes multiple semantic alignment modules; The step of fusing the first feature to be fused and the second feature to be fused based on the dynamic fusion weight to obtain fused features includes: Based on any of the fusion modules and the dynamic fusion weights in any of the fusion modules, feature fusion is performed on the first feature to be fused and the second feature to be fused output by any of the semantic alignment modules to obtain any multi-scale fusion feature; The fusion features are determined based on all the aforementioned multi-scale fusion features.

[0073] Specifically, the collaborative alignment module is not a single entity, but rather consists of multiple semantic alignment modules distributed across different feature scales. Similarly, the dynamic fusion module also consists of multiple fusion modules, each corresponding to a semantic alignment module.

[0074] Accordingly, based on any fusion module and the dynamic fusion weights in any fusion module, feature fusion can be performed on the first and second features to be fused output by any semantic alignment module to obtain any multi-scale fusion feature.

[0075] In other words, a hierarchical fusion mechanism is adopted. After acquiring multiple pairs of visible light modal features and infrared modal features from different levels during the feature extraction stage, semantic alignment and dynamic weighted fusion are performed independently for each scale. Specifically, for any given scale, the corresponding semantic alignment module first aligns the bimodal features at that scale, outputting a set of semantically consistent first and second features to be fused. The fusion module at that scale then generates fusion weights suitable for the features at the current scale based on its internal dynamic weight calculation mechanism, and completes feature fusion accordingly, generating multi-scale fused features at that scale. The above process can be executed in parallel or sequentially across scales, and finally, by aggregating the fused features from all scales, a comprehensive fused feature representation for target detection is formed.

[0076] This step is similar to the idea behind feature pyramid networks, which integrate multi-scale fusion features generated at all scales. For example, by upsampling or downsampling, these features are unified to the same resolution, and then element-wise addition or channel concatenation is performed to form a final fusion feature pyramid containing information from all scales. This fusion feature pyramid is then fed into the detection layer for final prediction.

[0077] The method provided in this invention extends the collaborative alignment and dynamic fusion mechanism to multiple scales, ensuring that the target detection model can perform effective modal information alignment and adaptive fusion regardless of the target size. This multi-scale processing architecture significantly enhances the target detection model's ability to detect targets of different sizes and improves the robustness of target detection.

[0078] Based on the above embodiments, the fusion module is used for: Based on the scene information of the first feature to be fused and the second feature to be fused received by the fusion module, the primary modal feature and the secondary modal feature in the first feature to be fused and the second feature to be fused are determined; The submodal features are fused into the main modal features to obtain the multi-scale fused features.

[0079] Specifically, when performing fusion, any fusion module is used for: Based on the scene information received by the fusion module from the first and second features to be fused, the dominant modality and submodal features in the first and second features to be fused are determined. At any scale, when the fusion module receives the first and second features to be fused corresponding to that scale, it first analyzes the scene information implied by these two features to determine which modality's information is more reliable at that scale. For example, the fusion module can analyze statistical quantities such as the overall brightness, contrast, and information entropy of the feature map. In well-lit scenes, features originating from visible light may be identified as dominant modal features; while in dark or smoky scenes, features originating from infrared light may be identified as dominant modal features. Correspondingly, the other is identified as a submodal feature.

[0080] Then, the submodal features can be fused into the main modal features to obtain multi-scale fused features.

[0081] After determining the primary and secondary modalities, the fusion process is not a simple symmetrical merging, but rather an asymmetric injection approach. Specifically, the fusion module integrates or injects supplementary information contained in the secondary modal features into the representation of the primary modal features. The advantage of this approach is that it preserves the complete structure and high-quality information of the dominant modal features while absorbing the unique advantages of the secondary modalities, avoiding the mutual contamination or information dilution that can occur when the two features are simply mixed. The resulting multi-scale fused features are essentially based on the primary modal features, enhanced and supplemented by the secondary modal features.

[0082] The method provided in this invention achieves adaptive multimodal feature fusion in the target detection model by introducing dynamic determination of primary and secondary modalities and an asymmetric fusion strategy. This mechanism dynamically determines and selects the most reliable modality as the fusion basis based on real-time input, and then enhances features using another modality. This method effectively improves the information quality and structural rationality of the fused features, thereby significantly enhancing the overall performance of the target detection model.

[0083] Based on the above embodiments, fusing the submodal features into the main modal features to obtain the multi-scale fused features includes: The main modal features, the intra-layer fusion features generated by the expert hybrid network, the mapped submodal features, the historical fusion results, and the static random noise features are concatenated to obtain concatenated features. The intra-layer fusion features are determined based on the first and second features to be fused received by the current fusion module. The mapped submodal features are obtained by performing dimensionality mapping on the submodal features in the current fusion module. The historical fusion results are the multi-scale fusion features output by the previous fusion module. The static random noise features are preset random numerical tensors. The splicing features are processed by the dimension transformation module to obtain the intermediate fused representation; The intermediate fusion representation is residually connected to the main modality feature to obtain the multi-scale fusion feature.

[0084] Specifically, firstly, the main modal features, the intra-layer fusion features generated by the expert hybrid network, the mapped submodal features, the historical fusion results, and the static random noise features are concatenated to obtain the concatenated features. The intra-layer fusion features are determined based on the first and second features to be fused received by the current fusion module. The mapped submodal features are obtained by performing dimensionality mapping on the submodal features in the current fusion module. The historical fusion results are the multi-scale fusion features output by the previous fusion module, and the static random noise features are preset random numerical tensors.

[0085] The intra-layer fusion feature is a preliminary result obtained by the expert hybrid network (MoE) within the current fusion module directly performing dynamic weighted fusion of the first and second input features to be fused. The intra-layer fusion feature represents the dynamic mixing of two modal information within the current scale.

[0086] Mapped submodal features refer to submodal features at the current scale that have been processed through a dimensional mapping layer to ensure that their number of channels is compatible with other features and facilitates subsequent splicing.

[0087] Historical fusion results refer to the output results passed from a fusion module at another scale. For example, in a top-down information flow, it is the multi-scale fusion feature from the previous level. This introduces cross-scale information interaction, allowing large-scale context to guide small-scale fusion.

[0088] Here, the static random noise feature is a pre-defined, fixed-value random tensor. Introducing this noise is a regularization technique that increases the randomness of the object detection model, helps prevent overfitting, and improves the generalization ability of the object detection model.

[0089] These five feature tensors, which come from different sources but have complementary functions, are concatenated along the channel dimension to form a concatenated feature with a very large number of channels.

[0090] Then, the concatenated features can be processed by a dimension transformation module to obtain an intermediate fused representation. The concatenated features contain redundant and highly complex information, requiring processing by a dimension transformation module. This module learns the complex interactions between the five information sources, extracts the most effective fusion information, and reduces the channel dimension of the features to a more reasonable level, thus obtaining the intermediate fused representation.

[0091] The dimension transformation module can be a module composed of several convolutional layers, activation functions and normalization layers, and this embodiment of the invention does not specifically limit it.

[0092] Finally, the intermediate fused representation is residually connected to the main modality features to obtain multi-scale fused features. Its core advantage lies in the fact that the object detection model only needs to learn incremental corrections to the main modality features, without needing to reconstruct the complete feature representation. This not only significantly improves the stability of the training process but also ensures that even when the contribution of the fusion module is limited, the system performance can still be guaranteed by reliable main modality features, thus maintaining the robustness of the overall detection performance.

[0093] Based on any of the above embodiments, a Weakly-aligned Multimodal Dynamic Hierarchical Fusion (WM-DHF) is proposed to address the semantic misalignment and heterogeneity issues in multimodal object detection. This method integrates two novel modules: Complementary Enhancement Feature Collaborative Alignment (CEF-CA) and Expert Hybrid Dynamic Hierarchical Fusion (DyFuse-MoE). The main contributions of this invention are as follows: (1) Design a Complementary-enhanced Feature Collaborative Alignment (CEF-CA) module to decompose multimodal features into modality-specific features and shared features, and achieve efficient alignment through attention mechanism and expert enhancement strategy; (2) Design a Dynamic Hierarchical Fusion with Mixture of Experts (DyFuse-MoE) module, adopt a progressive multi-stage fusion strategy, and use an expert mixing mechanism to dynamically guide the fusion of primary and secondary modal features according to scene conditions; (3) The Mixture of Experts (MoE) mechanism is deeply integrated into the CEF-CA and DyFuse-MoE modules. Through dynamic routing and fusion, the cross-modal feature representation capability is improved, thereby realizing adaptive modeling for different scenarios.

[0094] (4) The method of the present invention was tested using the VEDAI and DroneVehicle datasets. The results show that the present invention achieves a significant improvement in target detection accuracy compared with existing methods. Further ablation experiments verified the effectiveness of each module in improving feature alignment and fusion robustness.

[0095] Figure 3This is a schematic diagram of the MOE structure provided by the present invention. Figure 4 This is a schematic diagram of the dynamic layered fusion module provided by the present invention, as shown below. Figure 3 and Figure 4 As shown, the essential difference between multimodalities lies in the inherent modality-specific features of the semantic representation spaces of different modalities. They are different. This difference stems from their different physical perception mechanisms: for example, IR represents the thermal radiation temperature field, while RGB represents the reflectance spectral texture color. However, since both observe the same physical entity, their semantic spaces must share a semantic basis, that is, modal common features. Directly forcing the fusion of IR and RGB features will result in issues due to their modal-specific characteristics. The heterogeneity of features leads to modal conflicts. To address this issue, this study proposes a feature complementarity enhancement cooperative alignment module (CEF-CA), which consists of two core modules: cross-modal feature transfer alignment (CFTA) and shared feature enhancement based on hybrid experts (MoE-SFE).

[0096] 1) Preliminary decoupling of features: Preliminary definition of IR and RGB mode-specific features using the difference method. Given that each modal feature can be decomposed into unique features and common features, i.e. Based on this, the expression for the common characteristic components of each mode can be derived. However, the modality-specific features obtained by this heuristic method... and common features The semantic information represented is not precise enough and requires feature extraction through the following two modules.

[0097] 2) CFTA: This module pools the initial differential features using MP-Pool to generate descriptor vectors representing the importance of channels. These vectors are based on the fusion of global statistical characteristics and local salient responses, accurately quantifying the contribution of each channel to modality-specific information. Then, this attention weight is multiplied with the original modality input features channel by channel to achieve the purpose of adaptively filtering and enhancing core modality-specific information channels, ultimately extracting purer modality-specific features with stronger discriminative power.

[0098] 3) MoE-SFE: This module aims to refine the initial common features. It also enhances the common feature components within each mode. The MoE mechanism is used to improve... The representational power of MoE. MoE contains N expert networks with identical structures. Each expert is defined as Gated networks According to the input Dynamically calculate the weights of each expert .

[0099] in, It is a learnable, adaptive scaling weight. MoE focuses on ensemble learning. Different aspects, to obtain more accurate cross-modal shared features .

[0100] Then use precise Together with information from another modality, attention weights are generated through MP-Pool operations, and common feature components are extracted from and enhanced from the original features of this modality. 4) Collaborative Alignment and Fusion: Integrate the outputs of CFTA and MoE-SFE, and achieve cross-modal semantic space alignment through a dual-path strategy: introduce refined specific information from complementary modalities to promote alignment of specific semantic spaces; enhance common features to strengthen the shared semantic base within each modality.

[0101] in, These are learnable, adaptive weight coefficients used to optimize model performance. CEF-CA effectively mitigates modality conflicts and enhances the discriminative power of fused features through this collaborative mechanism, providing a more complete representation for downstream object detection tasks.

[0102] Here, in the Hybrid Expert Network (MoE), after receiving the input sample S, the gating network first performs a linear mapping on the input S to obtain a set of logits, and then performs Softmax normalization to output the weight coefficients k1…k corresponding to each expert network. n Instead of sending S to all expert networks (labeled E1, E2, ... Ei ... En), the gating network selects only a few (usually 1-2) expert networks with the highest weights for actual computation. Each selected expert network independently processes the same S and produces its output, while the remaining expert networks remain dormant during this forward propagation round. Subsequently, the gating network weights and sums the results of the selected expert networks according to their corresponding non-zero weights to obtain the final output of MoE. This approach ensures model capacity while significantly reducing computational overhead.

[0103] DyFuse-MoE is a dynamic hierarchical fusion framework based on hybrid experts, designed for weakly aligned multimodal scenarios. This method first distinguishes between primary and secondary modalities based on annotation information, establishing a "secondary-to-primary fusion" principle. The core principle is to preserve the advantages of the primary modality and ensure that the performance of the fused model is no lower than that of the original primary modality, thus avoiding the risk of performance regression. Its dynamic multi-layer architecture processes the fusion layer by layer: each layer uses historical fusion results as the primary input, with the initial layer containing the original features of the primary modality and fixed secondary modality representations as auxiliary inputs. DyFuse-MoE consists of N layers, where the first feature to be fused can be... The representation and intra-layer fusion features can be used The submodal features represented and mapped can be used The results of historical fusion can be represented as follows: This indicates that the characteristics of static random noise can be represented by... express.

[0104] Based on any of the above embodiments Figure 5 This is a schematic diagram of the target detection model provided by the present invention, as shown below. Figure 5 As shown, CEF-CA, as a weak alignment method, can promote semantic information interaction between multimodalities, but the original features of each modality still dominate. Given that the representations generated by this weak alignment are not completely equivalent, equivalence should be avoided in the design of multimodal fusion architectures. DyFuse-MoE is a dynamic hierarchical fusion framework based on hybrid experts, designed for weakly aligned multimodal scenarios. This method first distinguishes between primary and secondary modalities based on annotation information, establishing the principle of "secondary-to-primary fusion." The core is to retain the advantages of the primary modality and ensure that the performance of the fusion model is not lower than that of the original primary modality, avoiding the risk of performance regression. Its dynamic multi-layer architecture processes fusion layer by layer: each layer uses historical fusion results as the main input, where the initial layer contains the original features of the primary modality and fixes the secondary modality representation as auxiliary input. DyFuse-MoE includes N layers, namely... , , , and .

[0105] Intra-layer operations involve two key steps: 1) By concatenating five-element features through channels—the current primary modality input, the intra-layer fusion features generated by hybrid experts, the mapped secondary modality features, the previous layer fusion results, and static random noise—an intermediate fusion representation is generated through a dimension transformation module. 2) The intermediate result is supplemented to the main modality input using residual connections to form the final layer output. Features need to be preprocessed into a channel-dimensional dominant format to adapt to the core operations.

[0106] Intra-layer fusion is driven by a hybrid expert mechanism: after concatenating primary and secondary modal features, they are assigned to multiple experts through a learnable weight matrix. Each expert's weight matrix is ​​a learnable matrix, and after Softmax normalization, the weighted sum of the expert outputs is calculated. The core dynamism of this architecture lies in the fact that the current layer's fusion result includes the outputs of historical layers, and the outputs of historical layers depend on the hybrid expert results of their predecessor layers. Therefore, during training, updating the parameters of the current layer's experts through backpropagation will simultaneously optimize the parameters of previous layers, forming a dynamic fusion mechanism of cross-layer collaborative optimization. This makes the hybrid experts a key component in achieving the hierarchical fusion dynamism.

[0107] Furthermore, based on any fusion module and the dynamic fusion weights in any fusion module, feature fusion is performed on the first and second features to be fused output by any semantic alignment module to obtain any multi-scale fusion feature. Then, based on all multi-scale fusion features, the fusion feature is determined.

[0108] The target detection device based on multimodal dynamic fusion provided by the present invention will be described below. The target detection device based on multimodal dynamic fusion described below can be referred to in correspondence with the target detection method based on multimodal dynamic fusion described above.

[0109] Based on any of the above embodiments, the present invention provides a target detection device based on multimodal dynamic fusion. Figure 6 This is a schematic diagram of the target detection device based on multimodal dynamic fusion provided by the present invention, as shown below. Figure 6 As shown, the device includes: The acquisition unit 610 is used to acquire visible light images and infrared images, and to extract features from the visible light images and infrared images respectively to obtain visible light mode features and infrared mode features; The input unit 620 is used to input the visible light modal features and the infrared modal features into the target detection model to obtain the target detection result output by the target detection model; The target detection model includes a collaborative alignment module, a dynamic fusion module, and a detection layer; The collaborative alignment module is used to perform semantic alignment on the visible light modal features and the infrared modal features to obtain a first feature to be fused and a second feature to be fused. The dynamic fusion module includes an expert hybrid network, which is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain fused features; The detection layer is used to determine the target detection result based on the fusion features.

[0110] The apparatus provided in this invention comprises, on the one hand, a collaborative alignment module for semantically aligning visible light modal features and infrared modal features to obtain a first feature to be fused and a second feature to be fused, thereby effectively bridging the semantic differences between different modalities and reducing the interference of modality-specific noise on the expression of shared features; on the other hand, an expert hybrid network in the dynamic fusion module is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain fused features. This enables the target detection model to adaptively adjust its dependence on visible light and infrared modalities according to specific scenarios, avoiding the excessive dependence of static fusion strategies on failed modalities. Thus, while improving the quality of feature fusion, it significantly enhances the adaptability and reliability of the target detection model in complex and variable environments, ultimately ensuring the stability and robustness of detection accuracy.

[0111] Based on any of the above embodiments, the collaborative alignment module specifically includes: A decoupling module is used to decouple the visible light modal features and the infrared modal features to obtain modal shared features, visible light modal-specific features, and infrared modal-specific features; An enhancement module is used to enhance the modality-sharing features to obtain enhanced modality-sharing features; The purification module is used to purify the visible light mode-specific features and the infrared mode-specific features to obtain pure visible light mode-specific features and pure infrared mode-specific features; The compensation module is used to perform cross-modal information compensation based on the enhanced modal shared features, the pure visible light modal-specific features, and the pure infrared modal-specific features to obtain the first feature to be fused and the second feature to be fused.

[0112] Based on any of the above embodiments, the decoupling module is specifically used for: The mode-shared features are obtained by performing element-wise averaging on the visible light mode features and the infrared mode features; Perform a first element-wise difference operation on the visible light mode features and the infrared mode features to obtain the visible light mode-specific features; A second element-wise difference operation is performed on the infrared modal features and the visible light modal features to obtain the infrared modal-specific features.

[0113] Based on any of the above embodiments, the enhancement module is specifically used for: The modality sharing features are input into the gating network to obtain the routing weights corresponding to multiple expert networks output by the gating network; Based on the parallel processing of the modality-shared features by the multiple expert networks, multiple candidate enhanced shared features are obtained; The enhanced modality shared features are obtained by weighted summation of the multiple candidate enhanced shared features based on the routing weights.

[0114] Based on any of the above embodiments, the dynamic fusion module includes multiple fusion modules; the collaborative alignment module includes multiple semantic alignment modules; The dynamic fusion module is specifically used for: Based on any of the fusion modules and the dynamic fusion weights in any of the fusion modules, feature fusion is performed on the first feature to be fused and the second feature to be fused output by any of the semantic alignment modules to obtain any multi-scale fusion feature; The fusion features are determined based on all the aforementioned multi-scale fusion features.

[0115] Based on any of the above embodiments, the fusion module specifically includes: The primary and secondary modal feature determination module is used to determine the primary modal features and secondary modal features in the first and second features to be fused based on the scene information received by the fusion module. The fusion submodule is used to fuse the submodal features into the main modal features to obtain the multi-scale fused features.

[0116] Based on any of the above embodiments, the fusion submodule is specifically used for: The main modal features, the intra-layer fusion features generated by the expert hybrid network, the mapped submodal features, the historical fusion results, and the static random noise features are concatenated to obtain concatenated features. The intra-layer fusion features are determined based on the first and second features to be fused received by the current fusion module. The mapped submodal features are obtained by performing dimensionality mapping on the submodal features in the current fusion module. The historical fusion results are the multi-scale fusion features output by the previous fusion module. The static random noise features are preset random numerical tensors. The splicing features are processed by the dimension transformation module to obtain the intermediate fused representation; The intermediate fusion representation is residually connected to the main modality feature to obtain the multi-scale fusion feature.

[0117] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communications bus 740. The processor 710 can call logic instructions in the memory 730 to execute a target detection method based on multimodal dynamic fusion. This method includes: acquiring a visible light image and an infrared image, and extracting features from the visible light image and the infrared image respectively to obtain visible light modal features and infrared modal features; inputting the visible light modal features and the infrared modal features into a target detection model to obtain a target detection result output by the target detection model; the target detection model includes a collaborative alignment module, a dynamic fusion module, and a detection layer; the collaborative alignment module is used to perform semantic alignment on the visible light modal features and the infrared modal features to obtain a first feature to be fused and a second feature to be fused; the dynamic fusion module includes an expert hybrid network, which is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain a fused feature; the detection layer is used to determine the target detection result based on the fused feature.

[0118] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target detection method based on multimodal dynamic fusion provided by the above methods. The method includes: acquiring a visible light image and an infrared image, and extracting features from the visible light image and the infrared image respectively to obtain visible light modal features and infrared modal features; inputting the visible light modal features and the infrared modal features into a target detection model to obtain a target detection result output by the target detection model; the target detection model includes a collaborative alignment module, a dynamic fusion module, and a detection layer; the collaborative alignment module is used to perform semantic alignment on the visible light modal features and the infrared modal features to obtain a first feature to be fused and a second feature to be fused; the dynamic fusion module includes an expert hybrid network, which is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain a fused feature; the detection layer is used to determine the target detection result based on the fused feature.

[0120] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the target detection method based on multimodal dynamic fusion provided by the above methods. The method includes: acquiring a visible light image and an infrared image, and extracting features from the visible light image and the infrared image respectively to obtain visible light modal features and infrared modal features; inputting the visible light modal features and the infrared modal features into a target detection model to obtain a target detection result output by the target detection model; the target detection model includes a collaborative alignment module, a dynamic fusion module, and a detection layer; the collaborative alignment module is used to perform semantic alignment on the visible light modal features and the infrared modal features to obtain a first feature to be fused and a second feature to be fused; the dynamic fusion module includes an expert hybrid network, which is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain a fused feature; the detection layer is used to determine the target detection result based on the fused feature.

[0121] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection method based on multimodal dynamic fusion, characterized in that, include: Acquire visible light and infrared images, and extract features from the visible light and infrared images respectively to obtain visible light modal features and infrared modal features; The visible light modal features and the infrared modal features are input into the target detection model to obtain the target detection result output by the target detection model; The target detection model includes a collaborative alignment module, a dynamic fusion module, and a detection layer; The collaborative alignment module is used to perform semantic alignment on the visible light modal features and the infrared modal features to obtain a first feature to be fused and a second feature to be fused. The dynamic fusion module includes an expert hybrid network, which is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain fused features; The detection layer is used to determine the target detection result based on the fusion features.

2. The target detection method based on multimodal dynamic fusion according to claim 1, characterized in that, The collaborative alignment module is specifically used for: The visible light mode features and the infrared mode features are decoupled to obtain mode-shared features, visible light mode-specific features, and infrared mode-specific features; The modality sharing features are enhanced to obtain enhanced modality sharing features; The visible light mode-specific features and the infrared mode-specific features are purified to obtain pure visible light mode-specific features and pure infrared mode-specific features; Based on the enhanced modal sharing features, the pure visible light modal-specific features, and the pure infrared modal-specific features, cross-modal information compensation is performed to obtain the first feature to be fused and the second feature to be fused.

3. The target detection method based on multimodal dynamic fusion according to claim 2, characterized in that, The decoupling of the visible light mode features and the infrared mode features to obtain mode-shared features, visible light mode-specific features, and infrared mode-specific features includes: The mode-shared features are obtained by performing element-wise averaging on the visible light mode features and the infrared mode features; Perform a first element-wise difference operation on the visible light mode features and the infrared mode features to obtain the visible light mode-specific features; A second element-wise difference operation is performed on the infrared modal features and the visible light modal features to obtain the infrared modal-specific features.

4. The target detection method based on multimodal dynamic fusion according to claim 2, characterized in that, The enhancement process for the modality-sharing features to obtain enhanced modality-sharing features includes: The modality sharing features are input into the gating network to obtain the routing weights corresponding to multiple expert networks output by the gating network; Based on the parallel processing of the modality-shared features by the multiple expert networks, multiple candidate enhanced shared features are obtained; The enhanced modality shared features are obtained by weighted summation of the multiple candidate enhanced shared features based on the routing weights.

5. The target detection method based on multimodal dynamic fusion according to any one of claims 1 to 4, characterized in that, The dynamic fusion module includes multiple fusion modules; the collaborative alignment module includes multiple semantic alignment modules. The step of fusing the first feature to be fused and the second feature to be fused based on the dynamic fusion weight to obtain fused features includes: Based on any of the fusion modules and the dynamic fusion weights in any of the fusion modules, feature fusion is performed on the first feature to be fused and the second feature to be fused output by any of the semantic alignment modules to obtain any multi-scale fusion feature; The fusion features are determined based on all the aforementioned multi-scale fusion features.

6. The target detection method based on multimodal dynamic fusion according to claim 5, characterized in that, The fusion module is used for: Based on the scene information of the first feature to be fused and the second feature to be fused received by the fusion module, the primary modal feature and the secondary modal feature in the first feature to be fused and the second feature to be fused are determined; The submodal features are fused into the main modal features to obtain the multi-scale fused features.

7. The target detection method based on multimodal dynamic fusion according to claim 6, characterized in that, The process of fusing the submodal features into the primary modal features to obtain the multi-scale fused features includes: The main modal features, the intra-layer fusion features generated by the expert hybrid network, the mapped submodal features, the historical fusion results, and the static random noise features are concatenated to obtain concatenated features. The intra-layer fusion features are determined based on the first and second features to be fused received by the current fusion module. The mapped submodal features are obtained by performing dimensionality mapping on the submodal features in the current fusion module. The historical fusion results are the multi-scale fusion features output by the previous fusion module. The static random noise features are preset random numerical tensors. The splicing features are processed by the dimension transformation module to obtain the intermediate fused representation; The intermediate fusion representation is residually connected to the main modality feature to obtain the multi-scale fusion feature.

8. A target detection device based on multimodal dynamic fusion, characterized in that, include: The acquisition unit is used to acquire visible light images and infrared images, and to extract features from the visible light images and the infrared images respectively to obtain visible light mode features and infrared mode features; The input unit is used to input the visible light modal features and the infrared modal features into the target detection model to obtain the target detection result output by the target detection model; The target detection model includes a collaborative alignment module, a dynamic fusion module, and a detection layer; The collaborative alignment module is used to perform semantic alignment on the visible light modal features and the infrared modal features to obtain a first feature to be fused and a second feature to be fused. The dynamic fusion module includes an expert hybrid network, which is used to determine the dynamic fusion weights of the first feature to be fused and the second feature to be fused, and to perform feature fusion on the first feature to be fused and the second feature to be fused based on the dynamic fusion weights to obtain fused features; The detection layer is used to determine the target detection result based on the fusion features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the target detection method based on multimodal dynamic fusion as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the target detection method based on multimodal dynamic fusion as described in any one of claims 1 to 7.

Images