Vehicle target detection system and method based on mamba and dual domain interaction

Abstract

The present application relates to the technical field of image processing, and more particularly to a vehicle target detection system and method based on Mamba and double-domain interaction, wherein a backbone network adopts a double-flow Mamba structure to obtain multi-scale features of input visible light images and infrared images, in the process of scale extraction, the input features are scanned in multiple directions to enhance the structural detail representation ability required by the model for small target detection; a fusion network performs cross-modal information interaction on the visible light features and infrared features, through modal semantic enhancement and semantic alignment of different modalities in a deformable field, different modalities can be cooperatively represented; a neck network performs joint modeling of spatial and frequency domains on the fused features, realizes spatial-frequency domain enhancement, and makes spatial-frequency interaction more frequent, which is conducive to improving the detection capability for small targets. A head network extracts feature weights in two directions of height and width, while ensuring the model complexity, and also takes into account the accuracy, and outputs the category and position of the object.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a vehicle target detection system and method based on Mamba and dual-domain interaction. Background Technology

[0002] RGB-T multimodal remote sensing detectors have been widely used. Their greatest advantage lies in their ability to comprehensively utilize redundant information from visible light and infrared images. Due to their advantages such as all-weather operation, stable sensing, and high precision, RGB-T multimodal remote sensing detectors can be widely used in various scenarios such as intelligent transportation, security monitoring, autonomous driving, and military reconnaissance through intelligent remote sensing interpretation.

[0003] In recent years, deep learning has made rapid progress, especially in the field of computer vision, where a series of powerful architectures have achieved impressive performance. RGB-T multimodal detectors are modified from single-modal detectors. According to the modeling method, RGB-T multimodal detectors are divided into three categories: convolution-based multimodal modeling methods, Transformer-based multimodal modeling methods, and Mamba-based multimodal modeling methods.

[0004] However, RGB-T detection still faces many unresolved problems, mainly as follows: CNN or Transformer-based detection models suffer from insufficient global modeling or introduce secondary complexity leading to excessive computation; when fusing features from different modalities, structural and semantic misalignments can cause fuzziness in intermediate fusion features, affecting actual detection performance; remote sensing vehicles are small in size, and targets are easily obscured by the background, resulting in serious false positives and false negatives; since vehicles typically have rectangular or strip-shaped appearance features, many existing models often fail to utilize this geometric information. Summary of the Invention

[0005] In view of this, the present invention aims to provide a vehicle target detection system and method based on Mamba and dual-domain interaction. The backbone network acquires multi-scale features from the input visible light and infrared images. During the extraction process at each scale, it scans the input features in multiple directions, which enhances the structural detail representation capability required for small target detection. The fusion part performs cross-modal information interaction on the visible light and infrared features. Through modal semantic enhancement and semantic alignment of different modalities in deformable fields, different modalities can be collaboratively represented. The neck network performs joint spatial and frequency domain modeling of the fused features, achieving spatial-frequency domain enhancement and making spatial-frequency interaction more frequent, which is beneficial to improving the detection capability of small targets. The head network extracts feature weights in both height and width directions, ensuring both model complexity and accuracy, and outputs the object category and location.

[0006] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0007] A vehicle target detection system based on Mamba and dual-domain interaction includes: a backbone network employing a dual-stream Mamba structure to extract multi-scale features from input visible light and infrared images, obtaining multi-scale visible light and infrared features; and during the extraction process at each scale, the input features are scanned in multiple directions to ensure that the extracted features represent the structural details of the target; a fusion network performing cross-modal information interaction on visible light and infrared features of the same scale, and then performing semantic alignment and weighted fusion on the interacted visible light and infrared features to obtain the fused features at the current scale; and a neck network for processing multi-scale fused features. The system performs a pyramid fusion operation on the features, and then performs joint spatial and frequency domain modeling on the output features at each scale to obtain spatial-frequency domain fusion features, and introduces frequency domain compensation constraints to obtain frequency domain enhancement features. The spatial-frequency domain fusion features and frequency domain enhancement features are then fused to obtain spatial-frequency domain enhancement features at each scale. The head network extracts feature weights in both height and width directions for the spatial-frequency domain enhancement features at each scale, and performs cross-modal feature fusion on the multi-scale spatial-frequency domain enhancement features based on the extracted bidirectional feature weights to obtain detection features with the same number as the spatial-frequency domain enhancement features. The detection head is then used to perform target detection based on the corresponding detection features to obtain the target recognition results.

[0008] Furthermore, the backbone network includes two parallel and structurally consistent feature extraction branches; each feature extraction branch includes multiple consecutive feature extraction modules, each of which performs stepwise feature extraction on the input image or features, with each extracted feature corresponding to a scale; in each feature extraction module, depthwise separable convolution and multi-directional scanning operations are performed on the input features, and then the obtained features are concatenated with the input feature residuals to obtain intermediate features; after performing depthwise separable convolution on the intermediate features, they are added to the intermediate feature residuals, and the obtained features are multiplied by the intermediate feature residuals to obtain the output features of the feature extraction module.

[0009] Furthermore, the multi-directional scanning operation in the feature extraction module includes: scanning the input features along multiple directions, with the scanning results in each direction forming a feature sequence; inputting each feature sequence into a selective state-space model to model the complex interaction relationships existing in the long sequence, obtaining the corresponding enhanced feature sequence; merging the enhanced feature sequences corresponding to different directions to form an output feature of the same size as the input feature.

[0010] Furthermore, each feature extraction branch also includes multiple visual cue fusion modules. The input features at each scale are first processed by the visual cue fusion module before entering the feature extraction module for feature extraction processing.

[0011] Furthermore, the fusion network includes a cross-modal information interaction module and a semantic alignment module. In the cross-modal information interaction module: high-level and low-level features are extracted from visible light and infrared features of the same scale; corresponding weights are extracted from the four features, and then the weights are added to the corresponding features to obtain their respective enhanced features; the two enhanced features corresponding to the visible light features are fused to obtain fused visible light features; the two enhanced features corresponding to the infrared features are fused to obtain fused infrared features. In the semantic alignment module: the fused visible light features and fused infrared features are concatenated by channels, and the resulting concatenated features are divided into 4G+2 groups according to channels; the features in the first 2G groups of the concatenated features are visible light semantic features, and the features in the middle 2G groups are... Group G consists of infrared semantic features, while the last two groups consist of weighted mapping features. Convolution operations are performed on the visible light semantic features, infrared semantic features, and weighted mapping features respectively to complete the semantic offset mapping of the three features, resulting in visible light semantic offset fields, infrared semantic offset fields, and two modal-aware fusion weights. The visible light semantic offset fields and infrared semantic offset fields are then appended to the fused visible light features and fused infrared features respectively, resulting in visible light semantic offset vectors and infrared semantic offset vectors. Based on the visible light semantic offset vectors and infrared semantic offset vectors, the fused visible light features and fused infrared features are resampled respectively, and then the two resampled features are weighted and summed using the two modal-aware fusion weights to obtain the output fused features.

[0012] Furthermore, the process of obtaining enhanced features in the cross-modal information interaction module includes: concatenating and fusing the corresponding features of visible light and infrared features to obtain global features; performing global average pooling on the two features respectively to extract channel-level statistics to obtain their respective statistics; performing nonlinear mapping on the two statistics respectively through a shared parameter MLP to obtain their respective cross-modal weight perception coefficients; multiplying the two cross-modal weight perception coefficients with the global features and activating them with sigmoid, and then multiplying them with their respective corresponding features to obtain the enhanced features of the features in the same layer.

[0013] Furthermore, the neck network includes a feature pyramid module and a space-frequency interactive mixing module. The feature pyramid module performs bottom-up feature extraction at different levels from the input multi-scale fused features, as well as top-down high-level semantic information transfer, and finally fuses the high-semantic features at the same level with the high-resolution features at lower levels. The space-frequency interactive mixing module includes a spatial information extraction branch, a frequency domain information extraction branch, and a frequency domain compensation branch. In the spatial information extraction branch, depth convolution is performed on any scale feature output by the feature pyramid module to obtain spatial features. In the frequency domain information extraction branch, multi-directional frequency decomposition is performed on the spatial features. Then, the obtained multiple frequency bands are further processed... After the convolution operation, the processed frequency bands are integrated, and the integrated features are processed by a multilayer perceptron to obtain multi-directional adaptive weights. The multi-directional adaptive weights are applied to the spatial features, and then the resulting multi-directional features are calculated. The corresponding elements of the multi-directional features and spatial features are added together to obtain the spatial-frequency domain fusion features. In the frequency domain compensation branch, the features at any scale output by the feature pyramid module are transformed in the frequency domain, and the transformed frequency domain features are high-pass filtered. The multi-directional features are multiplied with the filtered frequency domain features and then transformed back to the spatial domain to obtain the frequency domain enhancement features. The spatial-frequency domain fusion features and the frequency domain enhancement features are fused together to obtain the spatial-frequency domain enhancement features at each scale.

[0014] Furthermore, the head network includes a pyramid and bidirectional fusion module, a scale unification module, and a detection head with the same number of spatial-frequency domain enhancement features. In the pyramid and bidirectional fusion module, a rectangular adaptive calibration operation is performed on the spatial-frequency domain enhancement features at each scale to obtain the calibration weights corresponding to each scale. In the scale unification module, for each scale of spatial-frequency domain enhancement feature, the scales of spatial-frequency domain enhancement features at other scales are transformed to the same scale as the current spatial-frequency domain enhancement feature. The transformed spatial-frequency domain enhancement features are then weighted and summed according to their respective calibration weights to obtain the detection features at the corresponding scale. The detection head performs target detection based on the corresponding detection features to obtain the target recognition result.

[0015] Furthermore, in the rectangular adaptive calibration operation: pooling operations are performed on the spatial frequency domain enhancement features in both height and width directions, and then the pooled features are added together to obtain the bidirectional fused features; sigmoid activation is performed on the bidirectional fused features, the result is multiplied with the corresponding elements of the input spatial frequency domain enhancement features, and then a fully connected operation is performed on the multiplied features to obtain the corresponding calibration weights.

[0016] A vehicle target detection method based on Mamba and two-domain interaction includes:

[0017] S1: Obtain a dataset including visible light images and corresponding infrared images, and perform target annotation on the images in the dataset to obtain target labels;

[0018] S2: Construct a vehicle target detection system based on Mamba and dual-domain interaction as provided in this invention;

[0019] S3: Using the images from the dataset in step S1 as input and the target labels as output, train the vehicle target detection system constructed in step S2 to obtain the vehicle target detection model;

[0020] S4: Input the visible light image to be detected and its corresponding infrared image into the vehicle target detection model obtained in step S3 to obtain the target vehicle detection result.

[0021] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0022] This invention presents a vehicle target detection system and method based on Mamba and dual-domain interaction. It employs a dual-Mamba structure as the backbone network, significantly reducing the computational complexity of the model and improving its utilization of global features. A cross-modal information interaction module and a semantic alignment module are designed in the fusion network. Through dual-modal semantic enhancement, semantic field prediction, and deformable differential alignment fusion, the structural misalignment problem between fusion modalities is effectively alleviated. The space-frequency interaction mixing module in the neck network comprehensively utilizes space-frequency dual-domain information, breaking through the limitations of the traditional spatial domain. Through space-frequency feature interaction, it improves the model's detection performance for small targets. The rectangular adaptive calibration operation in the head network utilizes the vehicle's elongated prior shape features and pyramid structure to enhance the model's utilization of vehicle prior features and improve detection accuracy. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0024] Figure 1 A schematic diagram of the vehicle target detection system based on Mamba and dual-domain interaction as described in an embodiment of the present invention;

[0025] Figure 2 A schematic diagram illustrating the CBS convolution operation described in an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of the feature extraction module described in an embodiment of the present invention;

[0027] Figure 4 A schematic diagram illustrating the multi-directional scanning operation described in an embodiment of the present invention;

[0028] Figure 5A schematic diagram of the cross-modal information interaction module described in an embodiment of the present invention;

[0029] Figure 6 A schematic diagram of the semantic alignment module described in an embodiment of the present invention;

[0030] Figure 7 A schematic diagram of the space-frequency interactive mixer module described in an embodiment of the present invention;

[0031] Figure 8 A schematic diagram illustrating the rectangular adaptive calibration operation described in an embodiment of the present invention;

[0032] Figure 9 A schematic flowchart of the vehicle target detection method based on Mamba and dual-domain interaction as described in the embodiments of the present invention;

[0033] Figure 10 The comparison results are shown in the embodiments of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0035] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more. Those skilled in the art can understand the specific meaning of the above terms in this invention through the specific circumstances.

[0036] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] like Figure 1As shown in the embodiment of the present invention, the vehicle target detection system based on Mamba and dual-domain interaction includes a backbone network, a fusion network, a neck network, and a head network. The backbone network adopts a dual-stream Mamba structure, performing multi-scale feature extraction on the input visible light image and infrared image respectively to obtain multi-scale visible light features and infrared features; and during the extraction process at each scale, the input features are scanned in multiple directions, so that the extracted features express the structural details of the target.

[0038] In some embodiments, the backbone network includes two parallel and structurally consistent feature extraction branches. Each feature extraction branch includes multiple consecutive feature extraction modules, each performing stepwise feature extraction on the input image or features, with each extracted feature corresponding to a scale. In some embodiments, each feature extraction branch also includes multiple visual cue fusion modules, where the input features at each scale are first processed by the visual cue fusion modules before entering the feature extraction modules for feature extraction processing.

[0039] In each feature extraction branch provided in the embodiments of the present invention, such as Figure 1 As shown, for an input image with a scale of H×W×C (i.e., an input visible light image or infrared image, where H represents the height, W represents the width, and C represents the number of channels), a series of CBS convolution operations and feature extraction operations from the feature extraction module are first performed to obtain a first-scale feature P1 with a scale of H / 2×W / 2×C. Then, it is processed sequentially through three feature extraction module groups consisting of a visual cue fusion module and a feature extraction module, resulting in a second-scale feature P2 with a scale of H / 4×W / 4×2C, a third-scale feature P3 with a scale of H / 8×W / 8×4C, and a fourth-scale feature P4 with a scale of H / 16×W / 16×8C. The CBS convolution operation is as follows: Figure 2 As shown, specifically, the input features are sequentially subjected to convolution, batch normalization, and SiLU activation before being concatenated with the input feature residuals.

[0040] This invention employs a dual-stream Mamba backbone architecture, introducing the State Space Model (SSM) into the field of object detection, and simultaneously designing a structured feature extraction module for object detection. SimpleStem is a crucial model component of Mamba, performing preliminary feature acquisition and transformation on the input image to reduce computational load and improve model detection efficiency through efficient dimensionality reduction. However, SimpleStem's two consecutive premature and rapid downsampling operations cause the model to irreversibly lose small target and edge structure feature information, thus reducing model accuracy. Therefore, this invention replaces the SimpleStem in the original Mamba backbone architecture with a more efficient feature extraction module. Object detection tasks typically involve high-resolution images or pixel sequences. Since the State Space Model was originally designed for modeling text sequences, it lacks the ability to fully utilize the channel depth information in images. To fully extract the enhanced details and multi-channel information provided by these high-resolution images, this invention proposes a feature extraction module to compensate for the shortcomings of the State Space Model in local modeling capabilities.

[0041] Specifically, in each feature extraction module, the input features are subjected to depthwise separable convolution and multi-directional scanning operations. The resulting features are then concatenated with the input feature residuals to obtain intermediate features. After performing depthwise separable convolution on the intermediate features, they are added to the intermediate feature residuals. The resulting features are then multiplied by the intermediate feature residuals to obtain the output features of the feature extraction module.

[0042] The feature extraction module provided in this embodiment of the invention is as follows: Figure 3 As shown, preprocessing the input features with CBS convolution helps adjust the dimension and distribution of the features and introduces non-linear activation, enabling the feature extraction module to learn more complex feature relationships. Borrowing from the Transformer Blocks architecture, layer normalization is applied to the features after CBS convolution to accelerate the overall training and convergence of the system. The above process can be represented by the following equation:

[0043] X1=LN(CBS(X in ))=LN(SiLU(BN(Conv(X in ))));

[0044] Among them, X in The input features of the feature extraction module are represented by CBS, the CBS convolution operation is represented by LN, the layer normalization process is represented by Conv, the convolution operation is represented by BN, the batch normalization operation is represented by X1, and the output features of the layer normalization process in the above operations are represented by X1.

[0045] The normalized output feature X1 is processed in parallel via two paths. In the first path, feature X1 undergoes a linear transformation, followed by depthwise separable convolution and SiLU activation. The processed feature is then subjected to a multi-directional scan (SS2D) operation and a linear transformation to obtain the output feature X2 of the first path. The above process can be expressed by the following formula:

[0046] X2=LR(SS2D(SiLU(DWConv(LR(X1)))));

[0047] Where DWConv represents depthwise separable convolution operation, and LR represents linear transformation;

[0048] In the second processing step, feature X1 undergoes a linear transformation and SiLU activation; the processed feature is then multiplied element-wise with the corresponding output feature X2 from the first processing step, followed by layer normalization, and finally multiplied with the input feature X. in Residual connection yields intermediate feature X3. The above process can be represented by the following formula:

[0049] X3=LN(X2×SiLU(LR(X1)))+X in ;

[0050] The intermediate feature X3 is subjected to layer normalization, and then the processed feature is processed in parallel via two paths: In the first path, the layer-normalized feature undergoes consecutive linear transformations, CBS convolutions, depthwise separable convolutions, and batch normalization, and is then concatenated with the feature residuals after the linear transformation; In the second path, the layer-normalized feature undergoes a linear transformation; The corresponding elements of the two processed features are multiplied together, and then sequentially subjected to linear transformations and CBS convolutions to obtain the final output feature X of the feature extraction module. out .

[0051] In the feature extraction module provided by this invention, two feature subspaces are split from the intermediate feature X3 by performing a bi-branch linear projection on the intermediate feature X3. In particular, the first processing uses linear transformation, CBS convolution operation, depthwise separable convolution, batch normalization processing, and residual connection process as a lightweight local feature texture model. Then, residual gradient fusion is used to add the original input through residual connection, so that the global dependency and features are passed to each pixel, making the model more sensitive to fine-grained features in the image, enhancing the model's expressive ability, and finally obtaining the output features.

[0052] In some embodiments, the multi-directional scanning operation in the feature extraction module includes: scanning the input features along multiple directions, with the scanning result of each direction forming a feature sequence; inputting each feature sequence into a selective state space model to model the complex interaction relationships existing in the long sequence, thereby obtaining the corresponding enhanced feature sequence; and merging the enhanced feature sequences corresponding to different directions to form an output feature with the same size as the input feature.

[0053] The multi-directional scanning operation (SS2D) processing procedure provided in this embodiment of the invention is as follows: Figure 4 As shown, a scanning expansion is first performed, that is, scanning is carried out from the diagonal view of the input features along the four symmetrical directions of the input features (from top to bottom, from bottom to top, from left to right, and from right to left). The scanning results in each direction form a feature sequence. This layout comprehensively covers all areas of the input image, providing a rich multidimensional information foundation for subsequent feature extraction, and enhancing the efficiency and comprehensiveness of multidimensional image feature capture.

[0054] Then, each feature sequence is input into a selective state-space model to model the complex interactions existing in the long sequence, resulting in the corresponding enhanced feature sequence. Specifically, the standard state-space model can be formalized into the following system of differential equations:

[0055] ;

[0056] Where x(t) represents the input signal of the state-space model at time t, h(t) represents the hidden state of the state-space model at time t, y(t) represents the output signal of the state-space model at time t, A represents the state transition matrix, B represents the input mapping matrix, and C and D represent direct transit terms. Let h(t) be the derivative of the hidden state with respect to time t.

[0057] When the input is a discrete sequence (such as a feature sequence), it is necessary to convert the input sequence {x1, x2, ..., x...} into a discrete sequence. K Discretize the continuous-time dynamical system and map it to the output sequence {y1, y2, ..., y}. K This process is called discretization modeling. Specifically, it includes:

[0058] Let the time step Δ be a predefined time scale parameter used to map the continuous state transition matrix A and the input mapping matrix B to the discrete space, as shown in the following equation:

[0059] ;

[0060] ;

[0061] in, This represents the discretized state transition matrix. Represents the discretized input mapping matrix;

[0062] At this point, the discrete hidden state h of the state-space model can be obtained. k and discrete output signal y K :

[0063] ;

[0064] in, Describe the discrete hidden state h at time k k The derivative with respect to time k; Represents the direct transitive terms of discretization; due to the direct transitive terms of discretization Essentially a residual connection, it does not affect the model's expressive power in actual implementation and can usually be ignored. Therefore, the output equation can be simplified as follows:

[0065] Y k =Ch k ;

[0066] When the time step Δ is small enough, the input mapping matrix A first-order Taylor expansion can be used as an approximation, which can effectively reduce the computational complexity caused by matrix exponentiation and inversion operations.

[0067] .

[0068] While state-space models are effective at modeling discrete sequences, they are insensitive to input variations, meaning their parameters remain constant regardless of input changes. To address this limitation, this invention introduces a selective state-space model (S. Mehta, M. Rastegari, "MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer", 2022.3.4, arXiv:2110.02178. doi: 10.48550 / arXiv.2110.02178), ensuring input dependency and enabling the model to be aware of the input context.

[0069] Finally, the processed feature sequences from different directions are concatenated to form an output image of the same size as the input image, thereby fusing the features extracted from different directions and realizing the extraction from local features to global features.

[0070] The fusion network performs cross-modal information interaction on visible light and infrared features of the same scale, and then performs semantic alignment and weighted fusion on the interacted visible light and infrared features to obtain the fused features at the current scale. In this embodiment of the invention, the second-scale feature P2, the third-scale feature P3, and the fourth-scale feature P4 of two images are specifically fused to obtain the second fused feature P2', the third fused feature P3', and the fourth fused feature P4'.

[0071] To explicitly model the coupling relationship between visible light (VI) and infrared (IR) features in the dual-domain semantics and alleviate the problem of semantic inconsistency in fused images caused by differences between modalities, this invention proposes a cross-modal information interaction module and a semantic alignment module. The two work together to achieve semantic synergy and fusion of different modal features through dual-modal semantic enhancement, learnable semantic flow field prediction, and deformable alignment fusion. At the same time, the cross-modal information interaction module and the semantic alignment module only introduce a small number of convolution and deformable sampling operators, with a computational complexity of only O(n), which is easy to embed into existing dual-modal detection networks and does not depend on a specific backbone structure.

[0072] In some embodiments, the fusion network includes a cross-modal information interaction module and a semantic alignment module. The cross-modal information interaction module can be viewed as a lightweight semantic enhancement attention module. Its function is to reduce modal differences between different scales and modalities, so that semantic flow prediction maintains geometric consistency as much as possible at different scales, thereby enhancing model robustness. Specifically, in the cross-modal information interaction module: high-level and low-level features are extracted from visible light features and infrared features of the same scale; corresponding weights are extracted from the four features respectively, and then the weights are attached to the corresponding features to obtain their respective enhanced features; the two enhanced features corresponding to the visible light features are fused to obtain fused visible light features; the two enhanced features corresponding to the infrared features are fused to obtain fused infrared features. In the semantic alignment module: the fused visible light features and fused infrared features are concatenated by channel, and the resulting concatenated features are divided into 4G+2 groups according to channels; the features in the first 2G groups of the concatenated features are visible light semantic features, the features in the middle 2G groups of the concatenated features are infrared semantic features, and the features in the last two groups are weighted mapping features; convolution operations are performed on the visible light semantic features, infrared semantic features, and weighted mapping features respectively to complete the semantic offset mapping of the three types of features, corresponding to the visible light semantic offset field, the infrared semantic offset field, and two modal-aware fusion weights; the visible light semantic offset field and the infrared semantic offset field are correspondingly appended to the fused visible light features and the fused infrared features, respectively, to obtain the visible light semantic offset vector and the infrared semantic offset vector; based on the visible light semantic offset vector and the infrared semantic offset vector, the fused visible light features and the fused infrared features are resampled respectively, and then the two resampled features are weighted and summed using the two modal-aware fusion weights to obtain the output fused features.

[0073] In some embodiments, the process of obtaining enhanced features in the cross-modal information interaction module includes: concatenating and fusing the corresponding features of visible light features and infrared features to obtain global features; performing global average pooling on the two features respectively to extract channel-level statistics to obtain their respective statistics; performing nonlinear mapping on the two statistics respectively through a shared parameter MLP to obtain their respective cross-modal weight perception coefficients; multiplying the two cross-modal weight perception coefficients with the global features and activating them with sigmoid, and then multiplying them with their respective corresponding features to obtain the enhanced features of the features in the same layer.

[0074] The cross-modal information interaction module provided in this embodiment of the invention is as follows: Figure 5 As shown, where: visible light features F of the same scale are... vi and infrared signature F ir As low-level features and The visible light feature F vi and infrared signature F ir Perform average pooling to obtain the corresponding high-level features. and Weights are extracted from the four features respectively, and then the weights are applied to the corresponding features to obtain the low-level features. and Enhancement features and and high-level characteristics and Corresponding enhancement features and The two enhancement features corresponding to visible light characteristics. and After fusion, the fused visible light features are obtained. The two enhancement features corresponding to the infrared features will be... and After fusion, fused infrared features are obtained. .

[0075] Specifically, two low-level features are obtained. and The corresponding enhancement feature process includes:

[0076] Low-level features corresponding to visible light and infrared features and The low-level global features G are obtained by first concatenating, then convolution, and finally SiLU activation. l As shown in the following formula:

[0077] ;

[0078] Where Cat represents channel splicing operation;

[0079] Two low-level features and Perform global average pooling separately to extract channel-level statistics and obtain their respective statistics. and :

[0080] , ;

[0081] Where P represents global average pooling;

[0082] Two statistics and The cross-modal weighted sensing coefficients are obtained by performing nonlinear mapping using a shared-parameter MLP (Multilayer Perceptron) according to the following formula. and :

[0083] , ;

[0084] Where σ represents sigmoid activation, ReLU represents the ReLU activation function, and W1 and W2 represent the shared weight matrices in the MLP;

[0085] The two cross-modal weight perception coefficients and With low-level global features G l After multiplication and sigmoid activation, each feature is multiplied by its corresponding feature in the same layer to obtain the enhanced features of that feature. and As shown in the following formula:

[0086] ;

[0087] .

[0088] Correspondingly, two high-level features are obtained. and Corresponding enhancement features and The process and two low-level features and Corresponding enhancement features and The process is the same, so I won't go into details here.

[0089] In this invention, fused visible light features are obtained. With fused infrared features The process is consistent with that of obtaining fused visible light features. For example, the process includes:

[0090] Characteristics corresponding to high-level Corresponding enhancement features Upsampling is performed to enhance the two types of features corresponding to visible light characteristics. and With consistent scales, the upsampled features are then activated by a sigmoid function and combined with the corresponding enhanced features from the lower layers. Multiplying corresponding elements yields the fused visible light features. The above process is as follows:

[0091] ;

[0092] Here, Up represents the upsampling operation. This embodiment of the invention utilizes high-level semantics to guide the selection of low-level details, introducing a feature fusion method with a semantic gating mechanism. This allows low-level features to focus on modally consistent regions under the guidance of high-level semantic features, achieving cross-scale semantic flow guidance.

[0093] The semantic alignment module provided in this embodiment of the invention is as follows: Figure 6 As shown, the obtained fused visible light features With fused infrared features Channel concatenation is performed, and the resulting concatenated features are divided into 4G+2 groups according to channels. The first 2G groups of concatenated features are visible light semantic features, the middle 2G groups are infrared semantic features, and the last two groups are weighted mapping features. Convolution operations are performed on the visible light semantic features, infrared semantic features, and weighted mapping features respectively to complete the semantic offset mapping of the three features, resulting in the visible light semantic offset field, the infrared semantic offset field, and the two modality-aware fusion weights, as shown in the following formula:

[0094] ;

[0095] in, and Representing the visible light semantic offset field and the infrared semantic offset field respectively, Att1 and Att2 represent the two modal sensing fusion weights, T θ Represents the semantic offset mapping implemented by the convolution operation;

[0096] The visible light semantic offset field and the infrared semantic offset field are correspondingly appended to the fused visible light feature and the fused infrared feature using the following formula, resulting in the visible light semantic offset vector and the infrared semantic offset vector:

[0097] , ;

[0098] in, This represents the visible light semantic offset vector corresponding to the g-th group of visible light semantic features. This represents the infrared semantic offset vector corresponding to the g-th group of infrared semantic features. This represents the visible light semantic offset field corresponding to the g-th group of visible light semantic features. Let (x, y) represent the infrared semantic offset field corresponding to the g-th group of infrared semantic features, and (x, y) represent the offset vector and the coordinates of the elements in the offset field.

[0099] Based on visible light semantic offset vectors and infrared semantic offset vectors, the fused visible light features are respectively... With fused infrared features Perform resampling as follows:

[0100] ;

[0101] ;

[0102] in, and This represents the fused visible light feature group (g-th) and the fused infrared feature group (g-th) after resampling. This indicates that the position (u,v) in the fused visible light features corresponds to the visible light semantic offset vector in the g-th group. Elements in the x-direction, This indicates that the position (u,v) in the fused visible light features corresponds to the visible light semantic offset vector in the g-th group. The element in the y-direction, This indicates that the position (u,v) in the fused infrared features corresponds to the infrared semantic offset vector in the g-th group. Elements in the x-direction, This indicates that the position (u,v) in the fused infrared features corresponds to the infrared semantic offset vector in the g-th group. In the y-direction, B represents the bilinear interpolation function;

[0103] The two resampled features are weighted and summed using two modality-aware fusion weights to obtain the output fused features, as shown in the following formula:

[0104] ;

[0105] in, This represents the fusion features of the output. and This represents the fused visible light features after resampling and the fused infrared features after resampling.

[0106] The neck network performs a pyramid fusion operation on the multi-scale fusion features, and then performs joint spatial and frequency domain modeling on the output features of each scale to obtain spatial-frequency domain fusion features, and introduces frequency domain compensation constraints to obtain frequency domain enhancement features; the spatial-frequency domain fusion features and frequency domain enhancement features are fused to obtain spatial-frequency domain enhancement features at each scale.

[0107] In some embodiments, the neck network includes a feature pyramid module and a space-frequency interactive mixing module. The feature pyramid module performs bottom-up feature extraction at different levels from the input multi-scale fusion features, and top-down high-level semantic information transfer, and finally fuses the high-semantic features at the same level with the high-resolution features at lower levels. In this embodiment, the feature pyramid module specifically fuses the second fusion feature P2', the third fusion feature P3', and the fourth fusion feature P4', corresponding to the processed second fusion feature P2'', the processed third fusion feature P3'', and the processed fourth fusion feature P4''. Then, the processed second fusion feature P2'', the processed third fusion feature P3'', and the processed fourth fusion feature P4'' are input into the space-frequency interactive mixing module, corresponding to the second space-frequency domain fusion feature P2''', the third space-frequency domain fusion feature P3''', and the fourth space-frequency domain fusion feature P4'''.

[0108] The space-frequency interactive mixing module unifies spatial structure modeling and frequency domain representation learning, addressing the low accuracy of small target detection caused by insufficient vehicle context modeling capabilities relying solely on spatial convolution. The module's mechanism involves applying joint space-frequency domain modeling and frequency domain compensation constraints to intermediate features. Specifically, the space-frequency interactive mixing module... Figure 7 As shown, it includes a spatial information extraction branch, a frequency domain information extraction branch, and a frequency domain compensation branch.

[0109] In the spatial information extraction branch, local feature modeling is performed by deep convolution on any scale feature output by the feature pyramid module. By allowing each channel to retain its own geometric structure pattern, spatial prior with structural fidelity is provided for the subsequent frequency branch, thus obtaining spatial features.

[0110] In the frequency domain information extraction branch, spatial features are decomposed into multiple directions using frequency decomposition. Specifically, wavelet transform is employed in this embodiment to map spatial features to low-frequency subbands (LL), horizontal high-frequency subbands (LH), vertical high-frequency subbands (HL), and diagonal high-frequency subbands (HH) at the spectral level. After convolution operations on the obtained frequency bands, the processed frequency bands are integrated, and the integrated features are processed by a multilayer perceptron to obtain multi-directional adaptive weights. In this embodiment, after performing 3×3 convolution on the four obtained frequency bands, a scale-module operation is also performed. After performing inverse wavelet transform on the four processed features to obtain integrated features, the diagonal high-frequency subbands are added to the integrated features. The added features are then processed by three consecutive multilayer perceptron (MLP) operations to obtain multi-directional adaptive weights. These multi-directional adaptive weights are applied to the spatial features, and the resulting multi-directional features are then added to the corresponding elements of the spatial features to obtain the spatial-frequency domain fusion features. The scale module operation is equivalent to a simplified learnable scaling operation, which specifically performs a learnable element-wise scaling of the input tensor (multiplying it by a trainable weight), and is equivalent to a parameterized multiplication operation.

[0111] In the frequency domain compensation branch, a frequency domain transformation is performed on any scale feature output by the feature pyramid module. In this embodiment of the invention, Fourier transform is used for the frequency domain transformation, and high-pass filtering is applied to the transformed frequency domain features. Specifically, Butterworth high-pass filtering is used in this embodiment of the invention:

[0112] ;

[0113] Where H(ω,υ) represents the Butterworth high-pass filter, (ω0,υ0) represents the center frequency, D0 represents the cutoff frequency, and n represents the order. The multi-directional features are multiplied by the filtered frequency domain features, and then converted back to the spatial domain using an inverse Fourier transform to obtain the frequency domain enhancement features. The spatial-frequency domain fusion features and the frequency domain enhancement features are fused by adding corresponding elements to obtain the spatial-frequency domain enhancement features at each scale.

[0114] The head network extracts feature weights in both height and width directions for the spatial-frequency domain enhancement features at each scale. Based on the extracted bidirectional feature weights, it performs cross-modal feature fusion on the multi-scale spatial-frequency domain enhancement features to obtain detection features with the same number as the spatial-frequency domain enhancement features. The detection head then performs target detection based on the corresponding detection features to obtain the target recognition result.

[0115] In some embodiments, the head network includes a pyramid and bidirectional fusion module, a scale unification module, and a detection head with the same number of spatial-frequency domain augmentation features. In the pyramid and bidirectional fusion module, a rectangular adaptive calibration operation is performed on the spatial-frequency domain augmentation features at each scale to obtain the calibration weights corresponding to each scale. In the scale unification module, for each scale of spatial-frequency domain augmentation feature, the scales of other scales of spatial-frequency domain augmentation features are transformed to the same scale as the current spatial-frequency domain augmentation feature. The transformed spatial-frequency domain augmentation features are then weighted and summed according to their respective calibration weights to obtain the detection features at the corresponding scale. The detection head performs target detection based on the corresponding detection features to obtain the target recognition result.

[0116] In some embodiments, the process of rectangular adaptive calibration is as follows: Figure 8 As shown, the method includes: performing pooling operations in both the height and width directions on the spatial frequency domain enhancement features to preserve structural dependencies and enhance the model's sensitivity to rectangular structures using prior knowledge of vehicle morphology; then, adding the pooled features to obtain bidirectional fused features. In this embodiment, both height and width directions are preserved simultaneously. The invention performs dual-domain collaborative fusion on the two pooled features using the following formula to achieve deep collaboration in the height-width direction within the spatial domain:

[0117] X rec =X h ×one 1×h +X w ×one 1×w ;

[0118] Among them, X rec Indicates the bidirectional fusion feature, X h Represents the features after height-oriented pooling, one 1×h The feature X is represented by the height-oriented pooling. h The matching all-one matrix, X w This represents the feature after pooling in the width direction, one 1×w The feature X after pooling in the width direction w Matching all-one matrices. Two all-one matrices are used to broadcast the features after two pooling to the original spatial size.

[0119] A sigmoid activation operation is performed on the bidirectional fused features. The result is multiplied element-wise with the corresponding element of the input space frequency enhancement features. A fully connected operation is then performed on the multiplied features to obtain the corresponding calibration weights. In this embodiment of the invention, the calibration weights are also normalized.

[0120] This invention also provides a vehicle target detection method based on Mamba and two-domain interaction, combined with Figure 1 and Figure 9 ,include:

[0121] S1: Obtain a dataset including visible light images and corresponding infrared images, and perform target annotation on the images in the dataset to obtain target labels.

[0122] In this embodiment of the invention, the visible light images and their matched infrared images are obtained from the DroneVehicle and VEDAI datasets. DroneVehicle is a multimodal vehicle detection dataset for traffic scenes from a drone's perspective, containing paired visible light and infrared images, suitable for multimodal fusion and target detection research. The data is collected from above by drones, covering various complex environments such as urban roads and parking lots, and including different lighting conditions such as daytime and nighttime. Compared to traditional datasets, the vehicle targets in this dataset are smaller in scale, have complex backgrounds, and exhibit significant differences in texture and response between the visible light and infrared modalities, increasing the difficulty of detection and fusion. The dataset provides annotations for five vehicle categories, such as cars, trucks, and vans, and has high research value. DroneVehicle is often used to evaluate the performance of infrared-visible light fusion algorithms and multimodal target detection methods, and is one of the important benchmark datasets in the current remote sensing detection field. VEDAI is a vehicle detection dataset for aerial remote sensing images, mainly used to study small target detection and fine-grained vehicle classification problems. This dataset consists of high-resolution aerial images covering various environments including urban, rural, and road scenes, exhibiting significant background complexity. VEDAI provides multi-category vehicle annotations, such as cars, trucks, truck trailers, and pickups, with detailed category classifications, facilitating refined recognition research. Due to the overhead perspective, vehicle targets are small and easily obstructed by background interference, thus placing higher demands on the robustness of the detection algorithms. This dataset is widely used in remote sensing target detection and UAV vision tasks.

[0123] S2: Construct a vehicle target detection system based on Mamba and dual-domain interaction as provided in this invention.

[0124] S3: Using the images from the dataset in step S1 as input and the target labels as output, train the vehicle target detection system constructed in step S2 to obtain the vehicle target detection model. In some embodiments, Cls loss and Reg loss are used to train the vehicle target detection system.

[0125] Furthermore, in this embodiment of the invention, the PyTorch deep learning framework is used and deployed and run on an Ubuntu 18.04 system. The SGD optimizer is used during training, and the batch size is set to 8.

[0126] S4: Input the visible light image to be detected and its corresponding infrared image into the vehicle target detection model obtained in step S3 to obtain the target vehicle detection result.

[0127] To verify that the method provided in this invention has good target vehicle detection performance, it was compared with existing single-modal detection methods (YOLOV11_m and RTDETRv2_pr34), existing Transformer-based multimodal detection methods (MRT-DETR_pr50), and existing Mamba-based multimodal detection methods (Fusion-Mamba and CFMW). The comparison test images were from the DroneVehicle dataset. The comparison results are as follows: Figure 10 As shown. From Figure 10 It can be seen that single-modal detection methods exhibit more significant false negatives and missed detections compared to multimodal detection methods, especially in the visible light modality, where this performance difference is even more pronounced. The main reason is that visible light images are highly susceptible to environmental factors such as changes in illumination, occlusion, and complex backgrounds, leading to unstable feature representation. In contrast, this invention, with the combined assistance of a cross-modal information interaction module and a semantic alignment module, effectively mitigates the feature differences between different modalities, enabling more thorough mining and fusion of multimodal information. Furthermore, compared to other multimodal methods, this invention demonstrates higher sensitivity and superior detection accuracy in detecting small-scale vehicle targets.

[0128] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A vehicle target detection system based on Mamba and dual-domain interaction, characterized in that, include: The backbone network adopts a dual-stream Mamba structure to extract multi-scale features from the input visible light and infrared images, respectively, to obtain multi-scale visible light and infrared features. Furthermore, during the extraction process at each scale, the input features are scanned in multiple directions, so that the extracted features express the structural details of the target. The fusion network performs cross-modal information interaction on visible light and infrared features of the same scale, and then performs semantic alignment and weighted fusion on the interacted visible light and infrared features to obtain the fused features at the current scale. The neck network performs pyramid fusion operation on multi-scale fusion features, and then performs joint spatial and frequency domain modeling on the output features of each scale to obtain spatial-frequency domain fusion features, and introduces frequency domain compensation constraints to obtain frequency domain enhanced features. By fusing spatial-frequency domain fusion features with frequency domain enhancement features, spatial-frequency domain enhancement features at each scale are obtained. The head network extracts feature weights in both height and width directions for the spatial-frequency domain enhancement features at each scale. Based on the extracted bidirectional feature weights, it performs cross-modal feature fusion on the multi-scale spatial-frequency domain enhancement features to obtain detection features with the same number as the spatial-frequency domain enhancement features. The detection head then uses the corresponding detection features to perform target detection and obtain the target recognition result. The converged network includes a cross-modal information interaction module and a semantic alignment module; In the cross-modal information interaction module: high-level and low-level features are extracted from visible light and infrared features of the same scale, respectively; The corresponding weights are extracted from the four features respectively, and then the weights are attached to the corresponding features to obtain their respective enhanced features; the two enhanced features corresponding to the visible light feature are fused to obtain the fused visible light feature. After fusing the two enhancement features corresponding to the infrared feature, the fused infrared feature is obtained; In the semantic alignment module: the fused visible light features and fused infrared features are concatenated by channel, and the resulting concatenated features are divided into 4G+2 groups according to channels; the features in the first 2G groups of the concatenated features are visible light semantic features, the features in the middle 2G groups of the concatenated features are infrared semantic features, and the features in the last two groups are weighted mapping features; convolution operations are performed on the visible light semantic features, infrared semantic features, and weighted mapping features respectively to complete the semantic offset mapping of the three features, corresponding to the visible light semantic offset field, the infrared semantic offset field, and two modal-aware fusion weights; the visible light semantic offset field and the infrared semantic offset field are correspondingly appended to the fused visible light features and the fused infrared features, respectively, to obtain the visible light semantic offset vector and the infrared semantic offset vector; based on the visible light semantic offset vector and the infrared semantic offset vector, the fused visible light features and the fused infrared features are resampled respectively, and then the two resampled features are weighted and summed using the two modal-aware fusion weights to obtain the output fused features; The neck network includes a feature pyramid module and a space-frequency interactive mixer module; The feature pyramid module performs bottom-up feature extraction at different levels on the input multi-scale fusion features, as well as top-down high-level semantic information transmission, and finally fuses the high-semantic features at the same level with the high-resolution features at the lower level. The space-frequency interactive mixing module includes a spatial information extraction branch, a frequency domain information extraction branch, and a frequency domain compensation branch. In the spatial information extraction branch, depthwise convolution is performed on any scale feature output by the feature pyramid module to obtain spatial features. In the frequency domain information extraction branch, multi-directional frequency decomposition is performed on the spatial features. After convolution operations on the obtained multiple frequency bands, the processed frequency bands are integrated, and the integrated features are processed by a multilayer perceptron to obtain multi-directional adaptive weights. These multi-directional adaptive weights are applied to the spatial features, and then multi-directional features are obtained. The corresponding elements of the multi-directional features and spatial features are added to obtain the space-frequency domain fusion features. In the frequency domain compensation branch, frequency domain transformation is performed on any scale feature output by the feature pyramid module, and high-pass filtering is applied to the transformed frequency domain features. The multi-directional features are multiplied by the filtered frequency domain features and then converted back to the spatial domain to obtain the frequency domain enhanced features. By fusing the spatial-frequency domain fusion features with the frequency domain enhancement features, spatial-frequency domain enhancement features at each scale are obtained.

2. The vehicle target detection system based on Mamba and dual-domain interaction according to claim 1, characterized in that, The backbone network consists of two parallel and structurally consistent feature extraction branches; Each feature extraction branch includes multiple consecutive feature extraction modules. Each feature extraction module performs step-by-step feature extraction on the input image or features, and each extracted feature corresponds to a scale. In each feature extraction module, the input features are subjected to depthwise separable convolution and multi-directional scanning operations. The resulting features are then concatenated with the input feature residuals to obtain intermediate features. After performing depthwise separable convolution on the intermediate features, they are added to the intermediate feature residuals. The resulting features are then multiplied by the intermediate feature residuals to obtain the output features of the feature extraction module.

3. The vehicle target detection system based on Mamba and dual-domain interaction according to claim 2, characterized in that, The multi-directional scanning operation in the feature extraction module includes: The input features are scanned along multiple directions, and the scan results in each direction form a feature sequence; Each feature sequence is input into a selective state-space model to model the complex interaction relationships that exist in long sequences, and the corresponding reinforcement feature sequence is obtained. The enhancement feature sequences corresponding to different directions are merged to form an output feature with the same size as the input feature.

4. The vehicle target detection system based on Mamba and dual-domain interaction according to claim 2, characterized in that, Each feature extraction branch also includes multiple visual cue fusion modules. The input features at each scale are first processed by the visual cue fusion module before entering the feature extraction module for feature extraction processing.

5. The vehicle target detection system based on Mamba and dual-domain interaction according to claim 1, characterized in that, The process of obtaining enhanced features in the cross-modal information interaction module includes: The corresponding features of visible light and infrared features in the same layer are spliced ​​and fused to obtain global features; Perform global average pooling on the two features at the same level and extract the channel-level statistics to obtain their respective statistics; The two statistics are nonlinearly mapped through a shared-parameter MLP to obtain their respective cross-modal weighted sensing coefficients. After multiplying the two cross-modal weighted perceptual coefficients with the global feature and activating them with the sigmoid function, they are then multiplied with their respective corresponding features in the same layer to obtain the enhanced features of each feature in the same layer.

6. The vehicle target detection system based on Mamba and dual-domain interaction according to claim 1, characterized in that, The head network includes a pyramid and bidirectional fusion module, a scale unification module, and a detection head with the same number of spatial frequency domain enhancement features; In the pyramid and bidirectional fusion module, a rectangular adaptive calibration operation is performed on the spatial frequency domain enhancement features at each scale to obtain the calibration weights corresponding to the spatial frequency domain enhancement features at each scale. In the scale unification module, for each scale of spatial frequency domain enhancement feature, the scale of spatial frequency domain enhancement features of other scales is transformed to the same scale as the current spatial frequency domain enhancement feature. The transformed spatial frequency domain enhancement features are then weighted and added according to their respective calibration weights to obtain the detection features of the corresponding scale. The detection head performs target detection based on the corresponding detection features to obtain the target recognition result.

7. The vehicle target detection system based on Mamba and dual-domain interaction according to claim 6, characterized in that, In rectangular adaptive calibration operation: Pooling operations are performed on the spatial frequency domain enhancement features in both height and width directions, and then the pooled features are added together to obtain the bidirectional fused features; A sigmoid activation operation is performed on the bidirectional fused features. The result is multiplied by the corresponding element of the input spatial-frequency domain enhancement features. A fully connected operation is then performed on the multiplied features to obtain the corresponding calibration weights.

8. A vehicle target detection method based on Mamba and dual-domain interaction, characterized in that, include: S1: Obtain a dataset including visible light images and corresponding infrared images, and perform target annotation on the images in the dataset to obtain target labels; S2: Construct a vehicle target detection system based on Mamba and dual-domain interaction as described in any one of claims 1 to 7; S3: Using the images from the dataset in step S1 as input and the target labels as output, train the vehicle target detection system constructed in step S2 to obtain the vehicle target detection model; S4: Input the visible light image to be detected and its corresponding infrared image into the vehicle target detection model obtained in step S3 to obtain the target vehicle detection result.

Images