A Small Target Detection Method and Device Based on Spatial-Frequency Co-driving Fusion

Abstract

This application relates to the field of small target detection technology, providing a method and apparatus for small target detection based on spatial-frequency collaborative fusion. This application utilizes an adaptive hybrid expert fusion module to fuse RGB features and Event features at both spatial location and frequency components, enabling bidirectional compensation of the spectral features of the two modes. This improves the effectiveness of small target detection performance under complex lighting conditions. The spatial fusion module generates spatial fusion features based on the difference and consistency responses of RGB and Event features at local locations. The generated spatial fusion features can adaptively allocate modal weights to different regions. The spectral fusion module generates spectral fusion features based on band masking, phase consistency, and modal energy proportions. The generated spectral fusion features can enhance the target-related frequency response supported by both modes within the effective frequency band, while suppressing noise exchange caused by low coherence or energy imbalance.

Description

Technical Field

[0001] This application relates to the field of small target detection technology, and in particular to a small target detection method and apparatus based on space-frequency collaborative driving fusion. Background Technology

[0002] Small target detection is an important task in computer vision, widely used in security and intelligent sensing scenarios. In practical applications, small targets are small in scale and lack texture, making them susceptible to interference from complex backgrounds, motion blur, and changes in lighting. Under extreme lighting conditions, traditional RGB cameras struggle to consistently capture clear images and boundaries of targets, and small moving targets are easily obscured by high dynamic range backgrounds, resulting in a significant decrease in detection performance.

[0003] Event cameras are asynchronous neuromorphic vision sensors that, with their microsecond-level temporal resolution, high dynamic range, and low latency, asynchronously output a sparse event stream composed of pixel positions, timestamps, and polarities. This effectively preserves edge and motion information in low-light, overexposed, and high-speed motion scenes, providing a new approach for small target detection in complex lighting conditions. However, the event stream lacks color and absolute brightness information, resulting in insufficient texture and semantic representation and limited detection accuracy when used alone. While RGB images possess rich texture and contextual information, they are prone to losing details under harsh conditions.

[0004] Existing RGB-Event fusion detection methods mostly employ feature concatenation, attention weighting, and cross-modal alignment, focusing on spatial domain interactions. However, for small object detection, the target region has few pixels, and its spatial features are easily affected by background and lighting changes. The spatial features of small objects are easily disturbed, and the robustness and accuracy of existing technologies for small object detection in complex scenes still need to be improved. Summary of the Invention

[0005] Therefore, it is necessary to provide a small target detection method and device based on space-frequency collaborative driving fusion to address the above-mentioned technical problems.

[0006] A small target detection method based on space-frequency collaborative fusion, the method comprising: Acquire the RGB image and Event image of the region where the small target to be detected is located, and extract the corresponding multi-scale RGB features and multi-scale Event features; In each feature layer, the RGB features and Event features are fused by an adaptive hybrid expert fusion module to obtain the output features of each feature layer. The adaptive hybrid expert fusion module includes a spatial fusion module and a spectrum fusion module. The spatial fusion module generates spatial fusion features based on the difference and consistency responses of the RGB features and Event features at local locations. The spectrum fusion module generates spectrum fusion features based on the frequency band mask, phase consistency, and modal energy ratio. The output features of each feature layer are input into the feature pyramid network and the detection head to obtain the detection results of small targets.

[0007] In one embodiment, at each feature layer, an adaptive hybrid expert fusion module fuses RGB features and Event features to obtain the output features of each feature layer, including: The spatial fusion module and the spectral fusion module are used to obtain the spatial fusion features and spectral fusion features of the current feature layer: ; ; in, It is a feature of spatial integration; For spectral fusion characteristics; This refers to the fusion operation of the spatial fusion module; This refers to the fusion operation of the spectrum fusion module; The RGB features of the current feature layer; The Event features of the current feature layer; The output features of the feature layer are obtained by aggregating the differences between the spatial fusion features and the spectral fusion features.

[0008] In one embodiment, the output features of the feature layer are obtained by aggregating the differences between the spatial fusion features and the spectral fusion features, including: Computational spatial fusion features and spectral fusion features: ; in, The difference in response between spatial fusion features and spectral fusion features; Extracting key region weights from differential responses using spatial attention: ; in, For attention maps; For spatial attention operations; Based on attention maps, a residual approach is used for expert aggregation: ; in, This is an aggregation feature; This is element-wise multiplication; use Convolution re-integrates the aggregated features through channels to obtain the output features of the feature layer: ; in, This refers to the output features of the feature layer.

[0009] In one embodiment, the spatial fusion feature is obtained according to the following steps: Simultaneously applying channel max pooling and channel average pooling to RGB and Event features, we obtain strong and average response information at each spatial location, thus acquiring the response information corresponding to the RGB and Event features: ; ; in, Max pooling of channels; Channel average pooling; For channel splicing; Response information for RGB features; Response information for Event features; Based on the response information corresponding to RGB features and Event features, calculate the difference response and the consistency response respectively: ; ; in, The response is a difference between the response information of RGB features and the response information of Event features; A consistent response; Will Concatenate along the channel dimension to generate input features for spatial weights, and then calculate the spatial weights: ; ; in, The input features for spatial weights; Spatial weights; It is a mapping function composed of convolutional layers and nonlinear activation functions; At each spatial location, the RGB weight value and Event weight value corresponding to the spatial weight are softmax normalized to obtain the spatial weight map of RGB and Event: ; in, Spatial weights for the RGB branches; Spatial weights for Event branches; Residual connections are used to weight RGB features and Event features: ; ; in, The weighted event features; The weighted RGB features; The weighted RGB features and the weighted Event features are concatenated along the channel dimension to obtain the spatial fusion feature: .

[0010] In one embodiment, the spectral fusion feature is obtained according to the following steps: A two-dimensional Fourier transform is performed on the RGB and Event features in the spatial dimension, mapping them to complex spectra: ; ; in, The complex spectrum corresponding to the RGB features; The complex spectrum corresponding to the event feature; It is a two-dimensional Fourier transform; Construct a learnable smooth bandpass mask and calculate the phase coherence between RGB and Event in the selected frequency band; Transform phase consistency into confidence weights for cross-modal spectral exchange: ; in, Confidence weights for cross-modal spectrum exchange; Ensure phase consistency between RGB and Event within the selected frequency band; based on Smooth bandpass masks and complex spectra to construct adaptive commutation coefficients: ; ; in, For adaptive swapping coefficients when aligning the Event spectrum to the RGB spectrum; For adaptive swapping coefficients when aligning the RGB spectrum to the Event spectrum; For smooth bandpass mask; Using soft spectrum alignment and Perform bidirectional cross-modal fusion: ; ; in, The complex spectrum corresponding to the fused RGB features; The complex spectrum corresponding to the fused Event features; The fused complex spectrum is mapped back to the spatial domain via inverse Fourier transform: ; ; in, The RGB features are enhanced with enhanced spectrum. Event features after spectrum enhancement; This is the inverse two-dimensional Fourier transform; The spectrally enhanced RGB features and the spectrally enhanced Event features are concatenated along the channel dimension to obtain the spectral fusion features: .

[0011] In one embodiment, the learnable smooth bandpass mask is: ; ; in, The radius is the frequency. For the Sigmoid function; Parameters used to control the smoothness of frequency band boundaries; For learnable low-frequency boundaries; For learnable high-frequency boundaries; The frequency coordinates in the horizontal direction of the two-dimensional spectrum plane; The frequency coordinates are in the vertical direction of the two-dimensional spectrum plane.

[0012] In one embodiment, calculating the phase coherence of RGB and Event within a selected frequency band includes: extract and Amplitude: ; ; in, for The amplitude; for The amplitude; Will and Normalized to unit complex form: ; ; in, This represents the complex spectrum corresponding to the normalized RGB features; The complex spectrum corresponding to the normalized Event features; Based on the normalized complex spectrum, obtain the phase consistency of RGB and Event at corresponding frequency positions: ; in, To ensure phase consistency between RGB and Event at corresponding frequency positions; This is a complex conjugate operation; To take the real part; By using an energy weighting mechanism, the phase consistency is statistically analyzed to obtain the phase consistency between RGB and Event within the selected frequency band: ; in, Ensures phase consistency between RGB and Event within the selected frequency band.

[0013] In one embodiment, the smooth bandpass mask employs a continuous transition at the band boundaries.

[0014] In one embodiment, the output features of each feature layer are input into a feature pyramid network and a detection head to obtain the detection results of small targets, including: The output features of each feature layer are input into the feature pyramid network, which outputs the fused pyramid features. The fused pyramid features are then input into the detection head to obtain the detection results for small targets.

[0015] A small target detection device based on space-frequency collaborative driving fusion, the device comprising: The data acquisition module is used to acquire the RGB image and Event image of the region where the small target to be detected is located, and extract the corresponding multi-scale RGB features and multi-scale Event features; The space-frequency driven fusion module is used to fuse RGB features and Event features at each feature layer through an adaptive hybrid expert fusion module to obtain the output features of each feature layer. The adaptive hybrid expert fusion module includes a spatial fusion module and a spectrum fusion module. The spatial fusion module generates spatial fusion features based on the difference and consistency responses of RGB features and Event features at local locations. The spectrum fusion module generates spectrum fusion features based on the frequency band mask, phase consistency, and modal energy ratio. The small target detection module is used to input the output features of each feature layer into the feature pyramid network and the detection head to obtain the detection results of small targets.

[0016] The aforementioned small target detection method and apparatus based on space-frequency collaborative fusion, through an adaptive hybrid expert fusion module, fuses RGB features and Event features at both spatial location and frequency component levels. This enables bidirectional compensation of the spectral features of the two modes (RGB and Event), generating space-frequency adaptive complementary output features, which effectively improves the performance of small target detection under complex lighting conditions. The spatial fusion module generates spatial fusion features based on the difference and consistency responses of RGB and Event features at local locations. The generated spatial fusion features can adaptively allocate modal weights to different regions, avoiding over-reliance on one type of response. The spectral fusion module generates spectral fusion features based on band mask, phase consistency, and modal energy ratio. The generated spectral fusion features can enhance the target-related frequency response supported by both modes within the effective frequency band, while suppressing noise exchange caused by low coherence or energy imbalance. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a small target detection method based on space-frequency collaborative fusion in one embodiment; Figure 2 This is a schematic diagram of the framework of a small target detection method based on space-frequency collaborative driving fusion in one embodiment; Figure 3 This is a flowchart illustrating the process of obtaining spatial fusion features in one embodiment; Figure 4 This is a flowchart illustrating the process of obtaining spectrum fusion features in one embodiment; Figure 5 This is a comparison chart of qualitative detection results of different methods on a dataset in one embodiment; Figure 6 This is a structural block diagram of a small target detection device based on space-frequency collaborative driving fusion in one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] In one embodiment, such as Figure 1 , Figure 2 As shown, a small target detection method based on space-frequency collaborative driving fusion is provided, the method comprising: Step 201: Obtain the RGB image and Event image of the region where the small target to be detected is located, and extract the corresponding multi-scale RGB features and multi-scale Event features.

[0020] It should be noted that the event images are acquired by the event camera. For the representation of the event stream, event frames are generated using a fixed time window and used as input for the event branches. These event frames allow the asynchronous event stream to be converted into a regular two-dimensional representation corresponding to the synchronous RGB image, facilitating subsequent dual-branch feature extraction and cross-modal fusion. The RGB images can be obtained by capturing them with a frame-based camera.

[0021] Step 202: In each feature layer, the RGB features and Event features are fused by an adaptive hybrid expert fusion module to obtain the output features of each feature layer. The adaptive hybrid expert fusion module includes a spatial fusion module and a spectrum fusion module. The spatial fusion module generates spatial fusion features based on the difference and consistency responses of the RGB features and Event features at local locations. The spectrum fusion module generates spectrum fusion features based on the frequency band mask, phase consistency, and modal energy ratio.

[0022] Step 203: Input the output features of each feature layer into the feature pyramid network and the detection head to obtain the detection results of small targets.

[0023] In the aforementioned small target detection method based on space-frequency collaborative fusion, an adaptive hybrid expert fusion module fuses RGB and Event features at both spatial location and frequency component levels. This enables bidirectional compensation of the spectral features of the two modes (RGB and Event), generating adaptively complementary space-frequency output features. This effectively improves the performance of small target detection under complex lighting conditions. The spatial fusion module generates spatial fusion features based on the difference and consistency responses of RGB and Event features at local locations. The generated spatial fusion features can adaptively allocate modal weights to different regions, avoiding over-reliance on one type of response. The spectral fusion module generates spectral fusion features based on band mask, phase consistency, and modal energy ratio. The generated spectral fusion features can enhance the target-related frequency response supported by both modes within the effective frequency band, while suppressing noise exchange caused by low coherence or energy imbalance.

[0024] In one embodiment, at each feature layer, an adaptive hybrid expert fusion module fuses RGB features and Event features to obtain the output features of each feature layer, including: The spatial fusion module and the spectral fusion module are used to obtain the spatial fusion features and spectral fusion features of the current feature layer: ; ; in, It is a feature of spatial integration; For spectral fusion characteristics; This refers to the fusion operation of the spatial fusion module; This refers to the fusion operation of the spectrum fusion module; The RGB features of the current feature layer; The Event features of the current feature layer; The output features of the feature layer are obtained by aggregating the differences between the spatial fusion features and the spectral fusion features.

[0025] In one embodiment, the output features of the feature layer are obtained by aggregating the differences between the spatial fusion features and the spectral fusion features, including: Computational spatial fusion features and spectral fusion features: ; in, The difference in response between spatial fusion features and spectral fusion features; It should be noted that the difference response is used to characterize the degree of inconsistency in the responses of the two types of fused features at different spatial locations. For small target areas, a larger difference response usually means that the current location may contain complementary information between the spatial and frequency domains.

[0026] Extracting key region weights from differential responses using spatial attention: ; in, For attention maps; For spatial attention operations; The generated attention map is used to highlight locations where the spatial fusion features and spectral fusion features differ significantly and may be more valuable for small target discrimination.

[0027] Based on attention maps, a residual approach is used for expert aggregation: ; in, This is an aggregation feature; This is element-wise multiplication; It should be noted that expert aggregation is based on As a baseline representation for aggregation, spatial fusion information is selectively introduced for aggregation under the guidance of differential attention. This approach preserves the main response of spectral fusion while introducing local discriminative cues provided by spatial experts, avoiding indiscriminate fusion of all regions. Since the spectral fusion features have already completed the bidirectional exchange of cross-modal frequency components through coherence constraints and energy balance, they can provide a relatively stable global spectral complementary representation. Meanwhile, the spatial fusion features focus more on modal reliability modulation at local locations and can serve as a supplement to key regions.

[0028] use Convolution re-integrates the aggregated features through channels to obtain the output features of the feature layer: ; in, This refers to the output features of the feature layer.

[0029] pass Convolution re-integrates aggregated features through channels, adjusting the information distribution between channels to make the output features more suitable for subsequent detection heads to perform target classification and bounding box regression.

[0030] In this embodiment, aggregation is performed based on the difference responses between spatial fusion features and spectral fusion features. This difference-guided aggregation reduces the weight of background clutter regions in the output features of the feature layer. It effectively utilizes the insensitivity of the spectral domain to background clutter to adjust the spatial domain, thereby reducing the impact of background clutter on small target detection and improving its accuracy. Furthermore, compared to direct addition or splicing, difference-guided aggregation introduces less redundant response and more effectively highlights complementary cues that are more effective for small target detection.

[0031] In one embodiment, such as Figure 3 As shown, the spatial fusion feature is obtained according to the following steps: Simultaneously applying channel max pooling and channel average pooling to RGB and Event features, we obtain strong and average response information at each spatial location, thus acquiring the response information corresponding to the RGB and Event features: ; ; in, Max pooling of channels; Channel average pooling; For channel splicing; Response information for RGB features; Response information for Event features; Based on the response information corresponding to RGB features and Event features, calculate the difference response and the consistency response respectively: ; ; in, The response information differs between RGB features and Event features. The larger the value, the greater the difference in the strength of the responses of the two modes at the current spatial location; A consistent response means that RGB and Event have the same response at the same spatial location. When both modalities have large response values ​​at the current spatial location, The larger the value, the weaker the response on one side. The value is relatively small; Will Concatenate along the channel dimension to generate input features for spatial weights, and then calculate the spatial weights: ; ; in, The input features for spatial weights; Spatial weights; It is a mapping function composed of convolutional layers and nonlinear activation functions; At each spatial location, the RGB weight value and Event weight value corresponding to the spatial weight are softmax normalized to obtain the spatial weight map of RGB and Event: ; in, Spatial weights for the RGB branches; Spatial weights for Event branches; Understandably, in each spatial location superior, Larger Corresponding to higher Event feature weights, larger This corresponds to higher RGB feature weights.

[0032] Residual connections are used to weight RGB features and Event features: ; ; in, The weighted event features; The weighted RGB features; The weighted RGB features and the weighted Event features are concatenated along the channel dimension to obtain the spatial fusion feature: .

[0033] In this embodiment, the location with a strong response on one side and a weak response on the other is marked by the difference response, enabling the weight generation process to focus on the local response of a small target that is only significant in a single modality; the location with a large response for both RGB and Event modalities is marked by the consistency response, so that when both modalities have responses in the same small target region or near a local boundary, the consistency response provides the basis for the consistent response location in the weight calculation. Together, they are used to adaptively generate spatial weights, enabling the weight generation function to simultaneously receive the original features, response differences, and common response information of both modalities, thus avoiding over-reliance on a single type of response for weight prediction.

[0034] In one embodiment, such as Figure 4 As shown, the spectrum fusion features are obtained according to the following steps: A two-dimensional Fourier transform is performed on the RGB and Event features in the spatial dimension, mapping them to complex spectra: ; ; in, The complex spectrum corresponding to the RGB features; The complex spectrum corresponding to the event feature; It is a two-dimensional Fourier transform; It should be noted that the complex spectrum consists of amplitude and phase. Amplitude represents the response intensity of each frequency component, and phase represents the positional relationship and structural distribution of the frequency components in the spatial domain.

[0035] Construct a learnable smooth bandpass mask and calculate the phase coherence between RGB and Event in the selected frequency band; Transform phase consistency into confidence weights for cross-modal spectral exchange: ; in, Confidence weights for cross-modal spectrum exchange; Ensure phase consistency between RGB and Event within the selected frequency band; It should be noted that negative coherent response usually indicates that there is phase inconsistency or spectral conflict between the two modes in the corresponding frequency band. Therefore, only positive coherent components are retained, and the distinction between high coherent response and low coherent response is further enhanced by squaring operation. Squaring operation can suppress weak coherent response, so that cross-mode exchange mainly occurs on the spectral components with stronger structural consistency. When the two modes are out of phase or conflict within the selected frequency band, the exchange confidence level is set to 0; when At that time, the exchange confidence level increases with increasing coherence.

[0036] based on Smooth bandpass masks and complex spectra to construct adaptive commutation coefficients: ; ; in, For adaptive swapping coefficients when aligning the Event spectrum to the RGB spectrum; For adaptive swapping coefficients when aligning the RGB spectrum to the Event spectrum; For smooth bandpass mask; It should be noted that, and This represents the modal energy percentage, used to measure the reliability of a mode at a corresponding frequency position, thereby avoiding excessive interference from low-energy or unstable frequency responses to another mode.

[0037] Using soft spectrum alignment and Perform bidirectional cross-modal fusion: ; ; in, The complex spectrum corresponding to the fused RGB features; The complex spectrum corresponding to the fused Event features; Understandably, bidirectional cross-modal fusion is not a simple superposition of modal spectra, but rather an adaptive soft alignment of the target modal spectrum within the reliable frequency band. When the exchange coefficient is small, the updated spectrum is closer to the target mode itself; when the exchange coefficient increases, the target mode is more fully aligned along the reliable modal spectrum direction. Furthermore, because... , The combined constraints of modal energy proportions enable the mapping process to enhance cross-modal complementary information while avoiding excessive amplification of unreliable frequency components and repetitive spectral responses.

[0038] The fused complex spectrum is mapped back to the spatial domain via inverse Fourier transform: ; ; in, The RGB features are enhanced with enhanced spectrum. Event features after spectrum enhancement; This is the inverse two-dimensional Fourier transform; The spectrally enhanced RGB features and the spectrally enhanced Event features are concatenated along the channel dimension to obtain the spectral fusion features: .

[0039] Understandably, the generated spectral fusion features can enhance the target-related frequency response supported by both modes within the effective frequency band, while suppressing noise exchange caused by low coherence or energy imbalance. They can complement the spatial fusion features and improve the discriminativeness and robustness of weak target representation from a spectral perspective.

[0040] In one embodiment, the learnable smooth bandpass mask is: ; ; in, The radius is the frequency. For the Sigmoid function; Parameters used to control the smoothness of frequency band boundaries; For learnable low-frequency boundaries; For learnable high-frequency boundaries; The frequency coordinates in the horizontal direction of the two-dimensional spectrum plane; The frequency coordinates are in the vertical direction of the two-dimensional spectrum plane.

[0041] It should be noted that, and It will be updated during training, enabling the adaptive hybrid expert fusion module to determine the frequency range to participate in spectrum exchange based on the detection task.

[0042] In one embodiment, the smooth bandpass mask employs a continuous transition at the band boundaries.

[0043] In this embodiment, the smooth bandpass mask uses a continuous transition at the frequency band boundary instead of directly using binary truncation, thereby avoiding abrupt changes in the mask value at the boundary and making the change in the spectrum exchange coefficient smoother.

[0044] In one embodiment, calculating the phase coherence of RGB and Event within a selected frequency band includes: extract and Amplitude: ; ; in, for The amplitude; for The amplitude; Will and Normalized to unit complex form: ; ; in, This represents the complex spectrum corresponding to the normalized RGB features; The complex spectrum is the normalized Event feature; phase information is preserved by normalizing the complex spectrum to a unit complex form.

[0045] Based on the normalized complex spectrum, obtain the phase consistency of RGB and Event at corresponding frequency positions: ; in, To ensure phase consistency between RGB and Event at corresponding frequency positions; This is a complex conjugate operation; To take the real part; when When the value is larger, it indicates that the two modes have a more consistent structural response at the current frequency components, and are more likely to correspond to target-related local changes; when When the value is small, it indicates a strong spectral conflict between the two modes, and the current frequency response may originate from background disturbances, event noise, or inconsistent modal information.

[0046] By using an energy weighting mechanism, the phase consistency is statistically analyzed to obtain the phase consistency between RGB and Event within the selected frequency band: ; in, Ensures phase consistency between RGB and Event within the selected frequency band.

[0047] Understandable. This represents the joint energy weight of RGB and Event. The energy weighting mechanism avoids interference from low-energy frequencies in coherence estimation. Frequency components with strong responses and phase consistency in both modes are given higher weight in coherence estimation, while the influence of low-energy or unstable frequency components is relatively weakened. It can more reliably reflect the degree of structural consistency between the two modes within the effective frequency band.

[0048] In one embodiment, the output features of each feature layer are input into a feature pyramid network and a detection head to obtain the detection results of small targets, including: The output features of each feature layer are input into the feature pyramid network, which outputs the fused pyramid features. The fused pyramid features are then input into the detection head to obtain the detection results for small targets.

[0049] Specifically, the feature pyramid network combines high-level semantic information with low-level spatial details through top-down information transmission and lateral connections, thereby enhancing the network's ability to represent small targets at different scales. The fused pyramid features are input into the detection head, which outputs the probability of target presence and its positional offset based on the features at each scale. After post-processing operations such as confidence filtering and non-maximum suppression, single-class target confidence prediction and bounding box regression are completed to obtain the detection results of small targets.

[0050] To verify the effectiveness and generalization ability of this application in the RGB-Event small target detection task, experiments were conducted on two publicly available RGB-Event detection datasets (RGBE-UAV and DSEC-MOD). RGBE-UAV was mainly used to evaluate the ability of this application to detect extremely small UAV targets under complex lighting conditions, while DSEC-MOD was used to verify the generalization performance of this application in moving target detection in autonomous driving scenarios.

[0051] The experiment employed a dual-branch ResNet-50 as the backbone network for both the RGB and event modalities, extracting multi-scale features from each modality respectively. Input data included RGB images and event images, with the event branch using event frames accumulated over a 30ms time window as input. Data preprocessing primarily involved normalization and resizing. During training, RGB images were randomly zeroed out with a 1% probability to enhance the robustness of the event modality. The model was trained using the Adam optimizer with an initial learning rate of 1×10⁻⁶. -4 The learning rate was dynamically adjusted using the ReduceLROnPlateau strategy, with a patience level of 5 (meaning that if the validation metrics show no significant improvement after 5 consecutive training epochs, the learning rate decays). The model was trained for 60 epochs by default, with a batch size of 8 per GPU. A gradient accumulation strategy was employed, updating parameters every 4 micro-batches, resulting in an equivalent batch size of 32. The gradient clipping threshold was set to 0.1. For the loss function, Focal Loss was used for the classification branch, with a class balance weight α = 0.25 and a hard sample focusing factor γ = 2.0. Smooth-L1 style regression loss was used for the bounding box regression branch. To ensure reproducibility, a fixed random seed of 42 was used, and CuDNN deterministic settings were enabled. All experiments were conducted on a single NVIDIA GeForce RTX 4090 GPU (24 GB), using PyTorch 2.0.1 and CUDA version 11.8. The evaluation metrics used are mAP, mAP50, and mAP75. The number of trainable parameters, FLOPs / GMACs, and FPS are also reported to comprehensively evaluate the detection accuracy and computational efficiency of the model.

[0052] To fairly evaluate the effectiveness of this application, the fusion module in this paper is compared with existing RGB-Event fusion methods under the same event representation, backbone network, detection head, and training settings, thus ensuring that the fusion module is the primary variable. Furthermore, several classic detection methods are introduced under RGB-only and Event-only settings to evaluate the detection performance of each single modality.

[0053] Table 1 presents the quantitative results on the RGBE-UAV dataset, where Pr represents precision, Re represents recall, and Fa represents false alarm rate.

[0054] Table 1. Quantitative comparison results of different detection methods on the RGBE-UAV dataset.

[0055] First, RGB-only methods generally outperform event-only methods, indicating that texture and semantic information in RGB images remain important for small object detection. However, single-modal methods have limited detection capabilities in complex lighting and weak object scenes. In contrast, RGB-Event fusion methods generally achieve better performance, demonstrating that motion edge information provided by event data can effectively supplement RGB representation.

[0056] Among all the comparison methods, this application achieves the best mAP50 and mAP75 on the RGBE-UAV dataset. Compared with existing RGB-Event fusion methods, this application shows a more significant improvement on the more stringent mAP75 metric, indicating that the Adaptive Hybrid Expert Fusion Module (AMoFE) can improve bounding box localization quality while enhancing target response. Furthermore, this application has fewer parameters than RENet, EOLO, and SGREF-Net while maintaining high inference speed, demonstrating a good balance between detection accuracy and computational efficiency.

[0057] Meanwhile, to verify the generalization ability of this application in other RGB-Event moving target detection scenarios, a dual-modal fusion comparison experiment was further conducted on the DSEC-MOD dataset. Table 2 shows the experimental results.

[0058] Table 2 Quantitative comparison results of RGB-Event fusion methods on the DSEC-MOD dataset

[0059] To further visualize and analyze the performance of this application in small target detection in complex scenes, the detection results of different methods are compared on the RGBE-UAV dataset, such as... Figure 5 As shown, Figure 5 In the diagram, green rectangles represent ground truth bounding boxes, red rectangles represent model prediction bounding boxes, and blue dashed arrows point to the area where the small target is located and its corresponding magnified local result.

[0060] from Figure 5 It can be seen that existing methods are prone to missed detections and false detections in scenarios with overexposure, underexposure, and complex backgrounds. Especially in locally magnified areas, some methods struggle to accurately distinguish small targets from background interference, leading to missing bounding boxes, positional shifts, or incorrect detections. This indicates that single spatial fusion or fixed fusion strategies still have certain limitations in representing small targets under complex lighting conditions.

[0061] In contrast, this application demonstrates more stable detection of small UAV targets and generates more accurate bounding boxes under various lighting conditions. Even when the target appearance in the RGB image is degraded or the event response is interfered with by noise, this application still maintains the target region response well. The results show that the proposed AMoFE module can effectively integrate complementary information from RGB and Event, enhance the discrimination features of small targets, and reduce false detections and false negatives caused by complex backgrounds.

[0062] To verify the roles of each component in AMoFE, ablation experiments were conducted on the RGBE-UAV dataset. The experiments used a direct splicing fusion model of FPN-Fusion (Feature Pyramid Network-Fusion) as the baseline. While maintaining consistency in the backbone network, detection head, training parameters, and data processing methods, spatial fusion modules (DCSF), spectral fusion modules (CEASF), and difference-guided expert aggregation modules (Agg) were added. The impact of each module on the detection results was analyzed by comparing the changes in mAP50, mAP75, accuracy, recall, and false alarm rate among different model variants.

[0063] Table 3 presents the ablation experimental results of the AMoFE module.

[0064] Table 3 Ablation experimental results of each component of AMoFE on the RGBE-UAV dataset.

[0065] Compared to the FPN-Fusion baseline, the addition of DCSF improved the model's detection results, indicating that the spatial weights calculated from the difference response and the consistency response can adjust the proportion of RGB and Event at different locations, preserving the local response of the stronger side in small target regions. The addition of CEASF also improved the model's detection results, demonstrating that learnable band selection, phase coherence gating, and energy balance exchange can limit the direct exchange of inconsistent frequency components and supplement the target-related spectral response. When both DCSF and CEASF are added, the model uses both the local response at spatial location and the target response at frequency components, further improving the detection results. Building upon this, the addition of a difference-guided expert aggregation module resulted in the best performance for the complete AMoFE, indicating that this aggregation module can integrate the two types of features based on the difference between the spatial fusion result and the spectral fusion result.

[0066] To further analyze the role of difference response (Diff) and consistency response (Cons) in DCSF, starting from the basic spatial fusion variant, difference information and consistency information were gradually added, and the detection results under different settings were compared. The results are shown in Table 4.

[0067] Table 4 Ablation experimental results of differential and uniform responses in DCSF

[0068] The base variant generates spatial weights solely using channel pooling results from RGB and Event features, without explicitly calculating the difference and consensus responses between the two modalities. Adding difference responses allows the model to capture the response differences between RGB and Event features at the same location, enabling spatial weight allocation to focus on small target local regions that are only significant within a single modality. Adding consensus responses allows the model to utilize the common responses of both modalities in the same target region, providing information on common activation locations for spatial weight generation. When both difference and consensus responses are added simultaneously, DCSF can utilize both the difference in bimodal responses and the common response locations, further improving detection results. These results demonstrate that both difference and consensus responses can provide effective information for spatial weight generation in DCSF.

[0069] To verify the role of each component in CEASF, we started with a basic spectrum fusion variant and gradually added learnable band selection, phase coherence gating, and energy balance mechanism. The detection results of different variants were compared, and the results are shown in Table 5.

[0070] Table 5 Ablation experimental results of CEASF mid-frequency band selection, phase gating and energy balance mechanism

[0071] The basic variant fuses RGB and Event features only in the frequency domain, without constraining the frequency range, phase consistency, or modal energy proportion involved in the exchange. By adding learnable frequency band selection, the module can determine the frequency range involved in spectrum exchange based on the training process, reducing the participation of frequency components with weak correlation to the target in the fusion. Further, by adding phase coherence gating, the module can control the exchange intensity based on the phase consistency of RGB and Event at corresponding frequency positions, reducing the direct participation of frequency components with large phase differences in the exchange. Finally, by adding an energy balancing mechanism, the module can adjust the exchange coefficients based on the energy proportion of the mode at corresponding frequency positions, reducing the impact of low-energy spectral responses on the other mode. The complete CEASF achieves better detection results, indicating that the above three designs collectively improve the spectrum fusion process.

[0072] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0073] In one embodiment, such as Figure 6 As shown, a small target detection device based on space-frequency cooperative driving fusion is provided, the device comprising: The data acquisition module 901 is used to acquire the RGB image and Event image of the region where the small target to be detected is located, and extract the corresponding multi-scale RGB features and multi-scale Event features.

[0074] The space-frequency driven fusion module 902 is used to perform feature fusion on RGB features and Event features at each feature layer through an adaptive hybrid expert fusion module to obtain the output features of each feature layer. The adaptive hybrid expert fusion module includes a spatial fusion module and a spectrum fusion module. The spatial fusion module generates spatial fusion features based on the difference and consistency responses of RGB features and Event features at local locations. The spectrum fusion module generates spectrum fusion features based on the frequency band mask, phase consistency, and modal energy ratio.

[0075] The small target detection module 903 is used to input the output features of each feature layer into the feature pyramid network and the detection head to obtain the detection results of small targets.

[0076] Specific limitations regarding the small target detection device based on space-frequency collaborative driving fusion can be found in the limitations of the small target detection method based on space-frequency collaborative driving fusion described above, and will not be repeated here. Each module in the aforementioned small target detection device based on space-frequency collaborative driving fusion can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A small target detection method based on space-frequency collaborative driving fusion, characterized in that, The method includes: Acquire the RGB image and Event image of the region where the small target to be detected is located, and extract the corresponding multi-scale RGB features and multi-scale Event features; In each feature layer, the RGB features and Event features are fused by an adaptive hybrid expert fusion module to obtain the output features of each feature layer. The adaptive hybrid expert fusion module includes a spatial fusion module and a spectrum fusion module. The spatial fusion module generates spatial fusion features based on the difference and consistency responses of the RGB features and Event features at local locations. The spectrum fusion module generates spectrum fusion features based on the frequency band mask, phase consistency, and modal energy ratio. The output features of each feature layer are input into the feature pyramid network and the detection head to obtain the detection results of small targets.

2. The small target detection method based on space-frequency collaborative driving fusion according to claim 1, characterized in that, At each feature layer, the RGB features and Event features are fused using an adaptive hybrid expert fusion module to obtain the output features of each feature layer, including: The spatial fusion module and the spectral fusion module are used to obtain the spatial fusion features and spectral fusion features of the current feature layer: in, It is a feature of spatial integration; For spectrum fusion characteristics; This refers to the fusion operation of the spatial fusion module; This refers to the fusion operation of the spectrum fusion module; The RGB features of the current feature layer; The Event features of the current feature layer; The output features of the feature layer are obtained by aggregating the differences between the spatial fusion features and the spectral fusion features.

3. The small target detection method based on space-frequency collaborative driving fusion according to claim 2, characterized in that, Aggregation is performed based on the difference responses between spatial fusion features and spectral fusion features to obtain the output features of the feature layer, including: Computational spatial fusion features and spectral fusion features: in, The difference in response between spatial fusion features and spectral fusion features; Extracting key region weights from differential responses using spatial attention: in, For attention maps; For spatial attention operations; Based on attention maps, a residual approach is used for expert aggregation: in, This is an aggregation feature; This is element-wise multiplication; use Convolution re-integrates the aggregated features through channels to obtain the output features of the feature layer: in, This refers to the output features of the feature layer.

4. The small target detection method based on space-frequency collaborative driving fusion according to claim 3, characterized in that, The spatial fusion feature is obtained according to the following steps: Simultaneously applying channel max pooling and channel average pooling to RGB and Event features, we obtain strong and average response information at each spatial location, thus acquiring the response information corresponding to the RGB and Event features: in, Max pooling of channels; Channel average pooling; For channel splicing; Response information for RGB features; Response information for Event features; Based on the response information corresponding to RGB features and Event features, calculate the difference response and the consistency response respectively: in, The response is a difference between the response information of RGB features and the response information of Event features; A consistent response; Will Concatenate along the channel dimension to generate input features for spatial weights, and then calculate the spatial weights: in, The input features for spatial weights; Spatial weights; It is a mapping function composed of convolutional layers and nonlinear activation functions; At each spatial location, the RGB weight value and Event weight value corresponding to the spatial weight are softmax normalized to obtain the spatial weight map of RGB and Event: in, Spatial weights for the RGB branches; Spatial weights for Event branches; Residual connections are used to weight RGB features and Event features: in, The weighted event features; The weighted RGB features; The weighted RGB features and the weighted Event features are concatenated along the channel dimension to obtain the spatial fusion feature: 。 5. The small target detection method based on space-frequency collaborative driving fusion according to claim 4, characterized in that, The spectral fusion features are obtained according to the following steps: A two-dimensional Fourier transform is performed on the RGB and Event features in the spatial dimension, mapping them to complex spectra: in, The complex spectrum corresponding to the RGB features; The complex spectrum corresponding to the event feature; It is a two-dimensional Fourier transform; Construct a learnable smooth bandpass mask and calculate the phase coherence between RGB and Event in the selected frequency band; Transform phase consistency into confidence weights for cross-modal spectral exchange: in, Confidence weights for cross-modal spectrum exchange; To ensure phase consistency between RGB and Event within the selected frequency band; based on Smooth bandpass masks and complex spectra to construct adaptive exchange coefficients: in, The adaptive swapping coefficients are used when aligning the Event spectrum to the RGB spectrum. For adaptive swapping coefficients when aligning the RGB spectrum to the Event spectrum; For smooth bandpass mask; Using soft spectrum alignment and Perform bidirectional cross-modal fusion: in, The complex spectrum corresponding to the fused RGB features; The complex spectrum corresponding to the fused Event features; The fused complex spectrum is mapped back to the spatial domain via inverse Fourier transform: in, The RGB features are enhanced with enhanced spectrum. Event features after spectrum enhancement; This is the inverse two-dimensional Fourier transform; The spectrally enhanced RGB features and the spectrally enhanced Event features are concatenated along the channel dimension to obtain the spectral fusion features: 。 6. The small target detection method based on space-frequency collaborative driving fusion according to claim 5, characterized in that, The learnable smooth bandpass mask is: in, The radius is the frequency. For the Sigmoid function; Parameters used to control the smoothness of frequency band boundaries; For learnable low-frequency boundaries; For learnable high-frequency boundaries; The frequency coordinates in the horizontal direction of the two-dimensional spectrum plane; The frequency coordinates are in the vertical direction of the two-dimensional spectrum plane.

7. The small target detection method based on space-frequency collaborative driving fusion according to claim 6, characterized in that, Calculate the phase consistency between RGB and Event within the selected frequency band, including: extract and Amplitude: in, for The amplitude; for The amplitude; Will and Normalized to unit complex form: in, This represents the complex spectrum corresponding to the normalized RGB features; The complex spectrum corresponding to the normalized Event features; Based on the normalized complex spectrum, obtain the phase consistency of RGB and Event at corresponding frequency positions: in, To ensure phase consistency between RGB and Event at corresponding frequency positions; This is a complex conjugate operation; To take the real part; By using an energy weighting mechanism, the phase consistency is statistically analyzed to obtain the phase consistency between RGB and Event within the selected frequency band: in, Ensures phase consistency between RGB and Event within the selected frequency band.

8. The small target detection method based on space-frequency collaborative driving fusion according to claim 6, characterized in that, Smooth bandpass masks employ continuous transitions at frequency band boundaries.

9. The small target detection method based on space-frequency collaborative driving fusion according to claim 7, characterized in that, The output features of each feature layer are input into the feature pyramid network and the detection head to obtain the detection results of small targets, including: The output features of each feature layer are input into the feature pyramid network, which outputs the fused pyramid features. The fused pyramid features are then input into the detection head to obtain the detection results for small targets.

10. A small target detection device based on space-frequency cooperative driving fusion, characterized in that, The device includes: The data acquisition module is used to acquire the RGB image and Event image of the region where the small target to be detected is located, and extract the corresponding multi-scale RGB features and multi-scale Event features; The space-frequency driven fusion module is used to fuse RGB features and Event features at each feature layer through an adaptive hybrid expert fusion module to obtain the output features of each feature layer. The adaptive hybrid expert fusion module includes a spatial fusion module and a spectrum fusion module. The spatial fusion module generates spatial fusion features based on the difference and consistency responses of RGB features and Event features at local locations. The spectrum fusion module generates spectrum fusion features based on the frequency band mask, phase consistency, and modal energy ratio. The small target detection module is used to input the output features of each feature layer into the feature pyramid network and the detection head to obtain the detection results of small targets.

Images