Visible light-event fused moving target detection method and system

Abstract

The invention discloses a visible light-event fusion moving target detection method and system. The method comprises the following steps: acquiring data and preprocessing the data; randomly initializing object query; updating the initial object query; extracting features; spatial perception feature positioning is completed; performing feature fusion on the fr and fe, and performing target detection according to the fused features to obtain a final detection result; the system comprises a data processing module, a random initialization object query module, an initial object query updating module, a feature extraction module and a spatial perception feature positioning module. According to the method, the potential moving objects in the image are mined by using the prior knowledge of the multi-modal large language model, and the position information of the target in the features is refined through the position information and the reference point for guiding the target detection, so that the feature learning is effectively guided.

Description

Technical Field

[0001] This invention relates to the field of moving target detection technology, specifically to a moving target detection method and system based on visible light-event fusion. Background Technology

[0002] Visible light sensor-based moving target detection has been widely explored and applied in fields such as autonomous driving and target tracking. Despite its success, it often suffers significant performance degradation when handling complex lighting conditions, such as nighttime or tunnel driving, due to environmental factors like illumination and darkness. This limitation has been a major factor restricting its performance. Therefore, researchers have attempted to enhance detection performance by introducing additional sensors. The complementarity and compatibility between event sensors and visible light sensors have facilitated the development of visible light-event moving target detection. This is largely due to the low latency of event sensors, which can quickly adapt to changes in illumination, making them more robust in low-light environments.

[0003] Most existing research on visible light-event moving target detection focuses on progressively merging event features to guide modeling. These works can be divided into two types: single-modal moving target detection and cross-modal moving target detection.

[0004] Single-modal modeling primarily extracts motion information of objects from images or event streams. While achieving seemingly reasonable results, these works are often limited, namely, they only cover visible light or events. The inherent limitations of a single modality are unavoidable. A single modality is easily subject to its own limitations, such as the rapid decline in detection capability under low light conditions and the lack of clear representation of target details in events. These limitations restrict the detection capability of single-modal moving target detection.

[0005] Compared with single-modal moving target detection, cross-modal moving target detection emphasizes complementary enhancement between different modalities, effectively avoiding the detection bottleneck inherent in single-modal methods. However, existing methods mainly achieve detection through simple feature splicing or accumulation fusion strategies, which are often unsuitable for different modalities. For example, direct fusion of sparse event modalities often introduces more noise, which will limit the performance of the model. Summary of the Invention

[0006] In view of this, the present invention provides a moving target detection method and system based on visible light-event fusion, in order to solve the problem that the extraction of position information of potential moving objects is not accurate enough in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A moving target detection method based on visible light-event fusion includes the following steps:

[0009] Acquire and preprocess the data: Select a historical dataset that includes visible light-event image pairs and preprocess the historical dataset;

[0010] Randomly initialize object queries: Perform random initialization for visible light and time object queries respectively, and obtain the visible light object query q. rgb And query the event object q event ;

[0011] Update initial object query: X the visible light image RGB The text input, along with its prompts, is fed into a multimodal large language model to obtain the prior position information F of the potential moving object. pos The prior location information F pos Visible light object lookup using cross-attention CA(·) rgb And query the event object q event The initialization yields q' respectively. rgb and q' event After feature mapping, the final visible light object query is obtained. and final event object query Use the final visible light object query Update q rgb Use the final event object to query Update q event ;

[0012] Feature extraction: Specifically targeting the visible light image X RGB and event image X EVENT Extract visible light features f rgb and event characteristics f event ;

[0013] Complete spatial awareness feature localization: Utilize the prior location information F output by the multimodal large language model... pos Decoding is performed to obtain the normalized coordinates B corresponding to the prior location information. After spatial weighting based on the normalized coordinates B, the final visible light feature f of the target area is obtained. r and the event characteristics of the final target area f e ;

[0014] f r f e , and The data is sent to the decoder, and the object is queried simultaneously. and The coordinates of the reference point of the potential moving object are obtained by linear projection. The decoder extracts the object feature f from the feature map based on the reference point coordinates. r and f e and complete the task for fr and f e Feature fusion is performed, and target detection is carried out based on the fused features to obtain the final detection result.

[0015] Preferably, the preprocessing includes cropping the image from the visible light-event image pair.

[0016] The preferred method for randomly initializing object queries is as follows:

[0017] q = Embedding(N,K)

[0018] Where q represents q rgb or q event Embedding(·) represents the embedding layer; N represents the number of queries; K represents the dimension of the query vector.

[0019] Preferably, the specific content of updating the initial object query includes:

[0020] X-ray visible light image RGB The predefined prompt text S is simultaneously input into a multimodal large language model (MLLM) to obtain prior location information:

[0021] F pos =MLLM(X RGB ,S)

[0022] in The text embedding is a position-encoded embedding, where L represents the number of words and D represents the dimension of each word;

[0023] The prior location information F pos Object lookup using cross-attention CA(·) q rgb and q event Initialization:

[0024] q′=CA(Linear(F pos ))

[0025] Where q' represents q' rgb or q' event Linear(·) represents a linear layer;

[0026] Feature mapping is performed by optimizing the feature information q' generated by cross-attention using a multilayer perceptron (MLP).

[0027]

[0028] in, express or LN(·) represents layer normalization.

[0029] Preferably, the specific content of completing spatial perception feature localization includes:

[0030] The output F of the multimodal large language model pos Decode the coordinates to obtain the normalized coordinates B:

[0031]

[0032] Among them, MLP decode (·) denotes a lightweight decoder, where d is an MLP. decode The token dimension of (·) This represents a decoder that maps d-dimensional real space to 4-dimensional real space.

[0033] By scaling the normalized coordinates B, a regional location information mask is obtained. and

[0034]

[0035] Where, x min x max x max y max The four locational information of B, through scale mapping, clarifies the region where the potential moving object is located. W and H represent visible light features f. rgb or event characteristics f event The width and height, where C represents the number of channels, i.e.

[0036] Generate a binary mask

[0037] Generate a single channel f rgb Full-size masked images of the same size, masked according to region location information. and Assigning values ​​to the full-1 mask image yields the corresponding binary mask M. Then, the pixel M in the i-th row and j-th column of the full-1 mask image... ij for:

[0038]

[0039] Using a Gaussian kernel G with a standard deviation of σ σ This is used to initialize the binary mask M, which helps to mitigate the boundary discontinuities of the hard mask.

[0040] M←G σ *M

[0041] Spatial weighting is used to suppress features in non-target regions and enhance the response within the region of interest (ROI):

[0042]

[0043] in, and These represent the latent moving object features that the multimodal large language model focuses on in the visible light and event modalities, respectively. ⊙ represents pointwise multiplication, and expand(M) copies M along the channel dimension to... OpticalFlow(·) represents an optical flow network, which consists of linear layers and is used to extract motion information from frames. λ is the motion weight coefficient.

[0044] Preferably, for f r and f e The process involves feature fusion and target detection based on the fused features. The final detection result includes the following:

[0045] f e and f r Perform concatenation and convolution operations to generate multi-scale fused features f':

[0046] f'=Cat(f e ,f r )

[0047] Where Cat(·) represents feature concatenation;

[0048] Inputting f' into the feedforward neural network outputs the prediction result:

[0049] Output = FFN(f')

[0050] Where FFN(·) represents a feedforward neural network, and Output represents the list of predicted candidate boxes output.

[0051] A moving target detection system based on visible light-event fusion using a multimodal large language model includes: a data processing module, a random initialization object query module, a latent moving object sensor, a feature extraction module, and a spatial perception feature localization module;

[0052] The data processing module is used to select historical datasets including visible light-event image pairs and to preprocess the historical datasets.

[0053] The random initialization object query module is used to randomly initialize visible light and time object queries separately, and obtain the visible light object query q. rgb And query the event object q event ;

[0054] A latent moving object perceptron, used to update the initial object query, includes a multimodal large language model, a cross-attention unit, and a feature mapping unit, wherein:

[0055] Multimodal large language model for use with visible light image X RGB The prior position information F of the potential moving object is obtained from the input of the prompt text. pos ;

[0056] Cross-attention units are used to process prior location information F pos Visible light object lookup using cross-attention CA(·) rgb And query the event object q event The initialization yields q' respectively. rgb and q' event ;

[0057] Feature mapping unit, used to map q' rg b and q' event The final visible light object query is obtained after feature mapping. and final event object query Use the final visible light object query Update q rgb Use the final event object to query Update q event ;

[0058] The feature extraction module includes a visible light feature extraction branch network and an event feature extraction branch network, which are used to extract features from visible light images X, respectively. RGB and event image X EVENT Extract visible light features f rgb and event characteristics f event ;

[0059] The spatially aware feature localization module includes a text decoder and a spatial weighting unit, wherein:

[0060] A text decoder is used to process the prior location information F output by a multimodal large language model. pos Decode the data to obtain the normalized coordinates B corresponding to the prior location information;

[0061] Spatial weighting cells are used to obtain the visible light characteristics f of the final target region after spatial weighting based on normalized coordinates B. r and the event characteristics of the final target area f e ;

[0062] The feature fusion module is used to combine f r f e , and The data is sent to the decoder, and the object is queried simultaneously. and The coordinates of the reference point of the potential moving object are obtained by linear projection. The decoder extracts the object feature f from the feature map based on the reference point coordinates. r and f e and complete the task for f r and f e Feature fusion is performed, and target detection is carried out based on the fused features to obtain the final detection result.

[0063] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a visible light-event fusion moving target detection method and system, which has the following beneficial effects:

[0064] This invention proposes a moving object detection method and system driven by a multimodal large language model, which mainly improves the limitations of traditional fusion strategies through a new object query-based paradigm. Specifically, this invention introduces a Latent Moving Object Perceptor (LMOP) and a Spatial Aware Feature Localization Module (SAFLM) to bridge the gap between position initialization and feature extraction. The LMOP guides the object query initialization using prior knowledge from the multimodal large language model. This design directs the reference point for object regression to the latent moving object, providing the model with the fastest and most accurate optimization direction. The SAFLM focuses on sampling effective object information, aiming to retain the most valuable information about the object. In summary, the LMOP and SAFLM work together to significantly enhance the feature representation of moving targets and provide richer position guidance for the decoder. This dual-module collaborative design can be easily integrated into existing detection frameworks, resulting in significant performance improvements. The semantic prior-based position guidance mechanism in this invention effectively focuses on key region features, avoids interference from irrelevant backgrounds, and facilitates the generation of more discriminative moving target feature representations. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a schematic flowchart of a visible light-event fusion moving target detection method disclosed in an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of the framework of a moving target detection system based on a multimodal large language model and visible light-event fusion, as disclosed in an embodiment of the present invention.

[0068] Figure 3 This is a schematic diagram illustrating the working principle of the potential moving object sensor or spatial sensing feature localization module disclosed in an embodiment of the present invention.

[0069] Figure 4 This is a comparison of the detection performance of MotionNet on the DSEC-MOD dataset. Figure 4 (a) is a comparison diagram of any detection result image of MotionNet on the DSEC-MOD dataset and the ground truth bounding box (GT). Figure 4 (b) is a comparison diagram of any other detection result image of MotionNet on the DSEC-MOD dataset with the ground truth bounding box (GT);

[0070] Figure 5 This is a comparison of the detection performance of MotionNet on the DELIVER-MOD dataset. Figure 5 (a) is a comparison diagram of any detection result image of MotionNet on the DELIVER-MOD dataset and the ground truth bounding box (GT). Figure 5 (b) is a comparison diagram of any other detection result image of MotionNet on the DELIVER-MOD dataset with the ground truth bounding box (GT). Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] This invention provides a moving target detection method based on visible light-event fusion, such as... Figure 1 As shown, it includes the following steps:

[0073] Acquire and preprocess the data: Select a historical dataset that includes visible light-event image pairs and preprocess the historical dataset;

[0074] Randomly initialize object queries: Perform random initialization for visible light and time object queries respectively, and obtain the visible light object query q. rgb And query the event object q event ;

[0075] Update initial object query: X the visible light image RGBThe text input, along with its prompts, is fed into a multimodal large language model to obtain the prior position information F of the potential moving object. pos The prior location information F pos Visible light object lookup using cross-attention CA(·) rgb And query the event object q event The initialization yields q' respectively. rg b and q' event After feature mapping, the final visible light object query is obtained. and final event object query Use the final visible light object query Update q rgb Use the final event object to query Update q event ;

[0076] Feature extraction: Specifically targeting the visible light image X RGB and event image X EVENT Extract visible light features f rgb and event characteristics f event ;

[0077] Complete spatial awareness feature localization: Utilize the prior location information F output by the multimodal large language model... pos Decoding is performed to obtain the normalized coordinates B corresponding to the prior location information. After spatial weighting based on the normalized coordinates B, the final visible light feature f of the target area is obtained. r and the event characteristics of the final target area f e ;

[0078] f r f e , and The data is sent to the decoder, and the object is queried simultaneously. and The coordinates of the reference point of the potential moving object are obtained by linear projection. The decoder extracts the object feature f from the feature map based on the reference point coordinates. r and f e and complete the task for f r and f e Feature fusion is performed, and target detection is carried out based on the fused features to obtain the final detection result.

[0079] It should be noted that:

[0080] In this embodiment, the DSEC-MOD and DELIVER-MOD datasets are selected. The DSEC-MOD dataset contains 16 sequences (13314 frames), and the DELIVER-MOD dataset contains 6280 frames, with each visible light image assigned a corresponding event image. As the model training and testing datasets, this invention divides the image pairs in the different datasets into training image pairs and testing image pairs to complete the training of each model. The training and testing sets of the DSEC-MOD dataset are divided in a 7:1 ratio, and the training and testing sets of the DELIVER-MOD dataset are divided in a 4:1 ratio.

[0081] In this implementation, feature extraction is performed through the feature extraction network in the end-to-end target detection model DTER.

[0082] The original DETR supports target detection by randomly initializing object queries. This randomization causes the model's initial focus to become divergent, leading it to incorrectly focus on unintended objects. Furthermore, DETR needs to shift its focus from divergence to accuracy, often resulting in higher training costs. To address these issues, this invention introduces a new branch to extract latent motion information from a multimodal large language model. By leveraging the prior knowledge of the multimodal large language model to obtain information about potential moving objects from images, this approach helps the model dynamically focus on target regions and effectively suppresses irrelevant background interference.

[0083] Furthermore, considering that extracting knowledge solely from complete images would lead to the extraction of more useless information (static objects), indirectly increasing the training time of the model and failing to fully utilize the prior information of the multimodal large language model to guide model training, this invention further introduces spatially perceptual feature localization. Spatially perceptual feature localization explores the alignment of the semantic information of the multimodal large language model with the spatial coordinates of visual features and guides the encoder features to migrate to predetermined potential motion positions.

[0084] In this embodiment, the prompt message of the present invention can be set to something like: "Please find all potential moving objects in this image and output the bounding boxes of the potential moving objects".

[0085] To further implement the above technical solution, preprocessing includes cropping the image in the visible light-event image pair.

[0086] It should be noted that:

[0087] In this embodiment, the input image is cropped to 640*480.

[0088] To further implement the above technical solution, the specific method for randomly initializing object queries is as follows:

[0089] q = Embedding(N,K)

[0090] Where q represents q rgb or q event Embedding(·) represents the embedding layer; N represents the number of queries; K represents the dimension of the query vector.

[0091] It should be noted that:

[0092] In this embodiment, N is 300 and K is 256.

[0093] To further implement the above technical solution, the specific content of the initial object query is updated as follows:

[0094] X-ray visible light image RGB The predefined prompt text S is simultaneously input into a multimodal large language model (MLLM) to obtain prior location information:

[0095] F pos =MLLM(X RGB ,S)

[0096] in The text embedding is a position-encoded embedding, where L represents the number of words and D represents the dimension of each word;

[0097] The prior location information F pos Object lookup using cross-attention CA(·) q rgb and q event Initialization:

[0098] q'=CA(Linear(F pos )

[0099] Where q' represents q' rgb or q' event Linear(·) represents a linear layer;

[0100] Feature mapping is performed by optimizing the feature information q' generated by cross-attention using a multilayer perceptron (MLP).

[0101]

[0102] in, express or LN(·) represents layer normalization.

[0103] It should be noted that:

[0104] By using an attention mechanism to capture long-distance dependencies and structural relationships between locations within a modality, q' can better describe object information in an image than randomly generated object queries q, thus providing more accurate query points for subsequent feature extraction and detection in DETR.

[0105] The feature information q' generated by cross-attention is optimized using a GELU-activated multilayer perceptron (MLP) to perform feature mapping and reduce the domain gaps between different features. Finally, the initial random query q is transferred to a new vector representation with detailed object location descriptions by leveraging the prior knowledge of a multimodal large language model.

[0106] In this embodiment, the prompt text S is set to "Please find all potential moving objects in this image" and the bounding boxes of the potential moving objects are output.

[0107] To further implement the above technical solution, the specific steps for completing spatial sensing feature localization include:

[0108] The output F of the multimodal large language model pos Decode the coordinates to obtain the normalized coordinates B:

[0109]

[0110] Among them, MLP decode (·) denotes a lightweight decoder, where d is an MLP. decode The token dimension of (·) This represents a decoder that maps d-dimensional real space to 4-dimensional real space.

[0111] By scaling the normalized coordinates B, a regional location information mask is obtained. and

[0112]

[0113] Where, x min x max x max y max The four locational information of B, through scale mapping, clarifies the region where the potential moving object is located. W and H represent visible light features f. rgb or event characteristics f event The width and height, where C represents the number of channels, i.e.

[0114] Generate a binary mask

[0115] Generate a single channel f rgbFull-size masked images of the same size, masked according to region location information. and Assigning values ​​to the full-1 mask image yields the corresponding binary mask M. Then, the pixel M in the i-th row and j-th column of the full-1 mask image... ij for:

[0116]

[0117] Using a Gaussian kernel G with a standard deviation of σ σ This is used to initialize the binary mask M, which helps to mitigate the boundary discontinuities of the hard mask.

[0118] M←G σ *M

[0119] Spatial weighting is used to suppress features in non-target regions and enhance the response within the ROI:

[0120]

[0121] in, and These represent the latent moving object features that the multimodal large language model focuses on in the visible light and event modalities, respectively. ⊙ represents pointwise multiplication, and expand(M) copies M along the channel dimension to... OpticalFlow(·) represents an optical flow network, which consists of linear layers and is used to extract motion information from frames. λ is the motion weight coefficient.

[0122] It should be noted that:

[0123] In this embodiment, B ∈ [0,1], which proportionally corresponds to the image size. MLP decode (·) mainly includes linear projection and Softmax, which transforms the dimension d into 4-dimensional boundary value parameters.

[0124] By using spatially perceptive feature localization, the problem of spatial alignment between visual encoder features and semantic cues from the MLLM multimodal large language model is effectively solved. The focus is shifted to the potential moving objects cued by the multimodal large language model, which helps the encoder dynamically focus on the target region and suppress irrelevant background interference, thereby improving the semantic perception and localization accuracy of multimodal tasks (such as video understanding, autonomous driving, etc.).

[0125] To further implement the above technical solution, for f r and f e The process involves feature fusion and target detection based on the fused features. The final detection result includes the following:

[0126] f e and f rPerform concatenation and convolution operations to generate multi-scale fused features f':

[0127] f'=Cat(f e ,f r )

[0128] Where Cat(·) represents feature concatenation;

[0129] Inputting f' into the feedforward neural network outputs the prediction result:

[0130] Output = FFN(f')

[0131] Where FFN(·) represents a feedforward neural network, and Output represents the list of predicted candidate boxes output.

[0132] A moving target detection system based on visible light-event fusion using a multimodal large language model, such as... Figures 2-3 As shown, it includes: a data processing module, a random initialization object query module, a potential moving object sensor, a feature extraction module, and a spatial perception feature localization module;

[0133] The data processing module is used to select historical datasets including visible light-event image pairs and to preprocess the historical datasets.

[0134] The random initialization object query module is used to randomly initialize visible light and time object queries separately, and obtain the visible light object query q. rgb And query the event object q event ;

[0135] A latent moving object perceptron, used to update the initial object query, includes a multimodal large language model, a cross-attention unit, and a feature mapping unit, wherein:

[0136] Multimodal large language model for use with visible light image X RGB The prior position information F of the potential moving object is obtained from the input of the prompt text. pos ;

[0137] Cross-attention units are used to process prior location information F pos Visible light object lookup using cross-attention CA(·) rg b and event object query q event The initialization yields q' respectively. rg b and q' event ;

[0138] Feature mapping unit, used to map q' rg b and q' event The final visible light object query is obtained after feature mapping. and final event object query Use the final visible light object query Update q rgb Use the final event object to query Update q event ;

[0139] The feature extraction module includes a visible light feature extraction branch network and an event feature extraction branch network, which are used to extract features from visible light images X, respectively. RGB and event image X EVENT Extract visible light features f rgb and event characteristics f event ;

[0140] The spatially aware feature localization module includes a text decoder and a spatial weighting unit, wherein:

[0141] A text decoder is used to process the prior location information F output by a multimodal large language model. pos Decode the data to obtain the normalized coordinates B corresponding to the prior location information;

[0142] Spatial weighting cells are used to obtain the visible light characteristics f of the final target region after spatial weighting based on normalized coordinates B. r and the event characteristics of the final target area f e ;

[0143] The feature fusion module is used to combine f r f e , and The data is sent to the decoder, and the object is queried simultaneously. and The coordinates of the reference point of the potential moving object are obtained by linear projection. The decoder extracts the object feature f from the feature map based on the reference point coordinates. r and f e and complete the task for f r and f e Feature fusion is performed, and target detection is carried out based on the fused features to obtain the final detection result.

[0144] The invention will be further explained below based on experiments:

[0145] This experiment designed four experimental configurations for system evaluation. Based on the technical solution disclosed in this invention, different components were used for object detection on the DSEC-MOD dataset. These included cases where either the latent moving object sensor or the spatial awareness feature localization module was used alone, or both were used simultaneously, or neither was used. Table 1 shows the performance comparison results for each case. The results show that the latent moving object sensor and the spatial awareness feature localization module can significantly improve model performance, with the overall performance index mAP being significantly higher. 50 The accuracy improved from a baseline of 59.5% to 62.45%. This improvement validates the effectiveness of the proposed module, which generates more discriminative motion feature representations by optimizing the feature learning process. Specifically, using the latent moving object perceptron alone can improve its mAP. 50 The performance improvement is 1.62%, while using the spatial awareness feature localization module alone brings a 1.81% performance gain. This improvement stems from the model's ability to deeply mine motion attributes: the latent moving object perceiver optimizes feature representation by capturing local motion direction information, while the spatial awareness feature localization module effectively reduces the complexity of moving target detection through spatial awareness mechanisms. After feature refinement, it can more accurately identify targets with motion attributes. Notably, the model achieves optimal performance (mAP) when both the latent moving object perceiver and the spatial awareness feature localization module are integrated. 50 =62.45%), this result fully demonstrates the rationality of the architecture design proposed in this invention. As shown in the theoretical analysis, the synergistic effect of the latent moving object sensor and the spatial perception feature localization module guides the generation of information features through a dual mechanism: the former enhances the discriminative power of motion features, and the latter optimizes the spatial feature representation, thus providing an innovative solution for moving target detection in visible light-event camera fusion scenarios.

[0146] Table 1 Applicability of different modules on the DSEC-MOD dataset

[0147]

[0148] Table 2 shows the effectiveness of motion attribute enhancement under different weights (λ = 0.1, 0.5, ...), illustrating the performance trend on the DSEC-MOD dataset. It can be seen that λ = 0.8 yields the best performance, with an mAP of 62.45%. This result demonstrates the importance of optical flow information enhancement in generating motion information, which can significantly affect detection performance. As λ increases from 0.8, the supplementation of optical flow information begins to weaken, and the performance drops from 62.45% to 61.84%. Conversely, decreasing λ also leads to performance degradation because the lack of intra-frame information limits detection performance, thus causing prediction errors.

[0149] Table 2 systematically evaluates the reference. Figures 2-3The impact of the mentioned structure on model performance under different motion enhancement weights (λ∈{0.1,0.5,0.8,1.0}) is discussed. Experimental data show that the model achieves optimal performance (mAP: 62.45%) on the DSEC-MOD dataset when λ=0.8, validating the crucial role of optical flow information in moving target detection. Notably, performance declines when λ>0.8 (e.g., mAP drops to 61.84% when λ=2.0), indicating that excessive motion feature enhancement disrupts the balance between appearance and motion features. Similarly, performance degrades when λ<0.8, due to insufficient motion information leading to missing temporal features. This phenomenon reveals an optimal coupling strength between appearance and motion features; moderate optical flow enhancement (λ=0.8) optimally supplements intra-frame motion cues, while over-reliance (λ↑) or neglect (λ↓) of motion information impairs detection performance.

[0150] Table 2 shows the performance trends on the DSEC-MOD dataset under different λ conditions.

[0151] Lambda 0.1 0.2 0.5 0.8 1.0 2.0 mAP 61.01％ 61.21％ 61.58％ 62.45％ 62.15％ 61.84％

[0152] Table 3 shows the performance improvement results of the proposed model on three different multimodal large language models. This invention first fixes the weight of λ to 0.8 to reduce randomness. Table 4 evaluates the performance of the three different multimodal large models, BLIP-3, QWen, and LLava, and reports the performance on DSEC-MOD. In the dataset test, LLava achieved the best performance, with an mAP improvement of 2.95, while BLIP-3 and QWen also improved by 2.45% and 2.54%, respectively. Clearly, the introduction of the multimodal large language model significantly improves the performance of moving object detection, demonstrating that its provided prior knowledge can effectively enhance the motion representation of objects and transfer the model's focus to other targets, thus providing more robust motion information.

[0153] Table 3 Performance Results

[0154]

[0155] Table 4 shows the performance comparison results of several multimodal detection methods on the DSEC-MOD and DELIVER-MOD datasets. The method proposed in this invention is denoted as MotionNet in the table. It can be seen that the performance of this invention in moving object detection is superior to other similar methods. Compared with the latest visible light-event moving object detection model RENet, this invention improves performance by 24.07%. This invention also compares with other visible light-event models, achieving significant improvements, which also proves the effectiveness of the method. Overall, event data is the main source for capturing motion information, while image data provides rich object feature information. The method proposed in this invention takes into account the differences between modalities and fully utilizes the prior knowledge of multimodal large language models to improve the detection process, refining the role of features of different potential moving objects, thereby guiding the model to learn effectively.

[0156] Table 4 Comparison Results

[0157]

[0158] This invention visualizes some of the detection results. Figures 4-5 This demonstrates the detection performance of MotionNet on the DSEC-MOD and DELIVER-MOD datasets, where Figure 4 (a) is a comparison diagram of any detection result image of MotionNet on the DSEC-MOD dataset and the ground truth bounding box (GT). Figure 4 (b) is a comparison diagram of any other detection result image of MotionNet on the DSEC-MOD dataset with the ground truth bounding box (GT); Figure 5 (a) is a comparison diagram of any detection result image of MotionNet on the DELIVER-MOD dataset and the ground truth bounding box (GT). Figure 5 (b) is a schematic diagram comparing any other detection result image of MotionNet on the DELIVER-MOD dataset with the ground truth bounding box (GT). All the targets reported in the image were correctly captured, with no missed or over-labeled objects. It can be seen that, using the method of this invention, the detection results are basically consistent with the labels.

[0159] In some alternative embodiments, the embodiments presented and described in the steps of the invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented in the invention. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are performed independently.

[0160] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A moving target detection method based on visible light-event fusion, characterized in that, Includes the following steps: Acquire and preprocess the data: Select a historical dataset that includes visible light-event image pairs and preprocess the historical dataset; Randomly initialize object queries: Perform random initialization for visible light and time object queries respectively, and obtain the visible light object query q. rgb And query the event object q event ; Update initial object query: X the visible light image RGB The text input, along with its prompts, is fed into a multimodal large language model to obtain the prior position information F of the potential moving object. pos The prior location information F pos Visible light object lookup using cross-attention CA(·) rgb And query the event object q event The reinitialization yields q' respectively. rg b and q' event After feature mapping, the final visible light object query is obtained. and final event object query Use the final visible light object query Update q rgb Use the final event object to query Update q event ; feature Extraction: For visible light image X RGB and event image X EVENT Extract visible light features f rgb and event characteristics f event ; Complete spatial awareness feature localization: Utilize the prior location information F output by the multimodal large language model... pos Decoding is performed to obtain the normalized coordinates B corresponding to the prior location information. After spatial weighting based on the normalized coordinates B, the final visible light feature f of the target area is obtained. r and the event characteristics of the final target area f e ; f r f e , and The data is sent to the decoder, and the object is queried simultaneously. and The coordinates of the reference point of the potential moving object are obtained by linear projection. The decoder extracts the object feature f from the feature map based on the reference point coordinates. r and f e and complete the task for f r and f e Feature fusion is performed, and target detection is carried out based on the fused features to obtain the final detection result.

2. The moving target detection method based on visible light-event fusion according to claim 1, characterized in that, Preprocessing includes: Cropping the image from the visible light-event image pair.

3. The moving target detection method based on visible light-event fusion according to claim 1, characterized in that, The specific method for randomly initializing object queries is as follows: q = Embedding(N,K) Where q represents q rgb or q event Embedding(·) represents the embedding layer; N represents the number of queries; K represents the dimension of the query vector.

4. The moving target detection method based on visible light-event fusion according to claim 1, characterized in that, The specific content of updating the initial object query includes: X-ray visible light image RGB The predefined prompt text S is simultaneously input into a multimodal large language model (MLLM) to obtain prior location information: F pos =MLLM(X RGB ,S) in This is a position-encoded text embedding, where L represents the number of words and D represents the dimension of each word. It is the set of real numbers; The prior location information F pos Object lookup using cross-attention CA(·) q rgb and q event Initialization: q′=CA(Linear(F pos )) Where q' represents q' rgb or q' event Linear(·) represents a linear layer; Feature mapping is performed by optimizing the feature information q' generated by cross-attention using a multilayer perceptron (MLP). in, express or LN(·) represents layer normalization.

5. The moving target detection method based on visible light-event fusion according to claim 1, characterized in that, The specific content of completing spatial perception feature localization includes: The output F of the multimodal large language model pos Decode the coordinates to obtain the normalized coordinates B: Among them, MLP decode (·) denotes a lightweight decoder, where d is an MLP. decode The token dimension of (·) This represents a decoder that maps d-dimensional real space to 4-dimensional real space. By scaling the normalized coordinates B, a regional location information mask is obtained. and Where, x min x max x max y max The four locational information of B, through scale mapping, clarifies the region where the potential moving object is located. W and H represent visible light features f. rgb or event characteristics f event The width and height, where C represents the number of channels, i.e. It is the set of real numbers; Generate a binary mask Generate a single channel f rgb Full-size masked images of the same size, masked according to region location information. and Assigning values ​​to the full-1 mask image yields the corresponding binary mask M. Then, the pixel M in the i-th row and j-th column of the full-1 mask image... ij for: Using a Gaussian kernel G with a standard deviation of σ σ This is used to initialize the binary mask M, which helps to mitigate the boundary discontinuities of the hard mask. M←G σ *M Spatial weighting is used to suppress features in non-target regions and enhance the response within the region of interest (ROI): in, and These represent the latent moving object features that the multimodal large language model focuses on in the visible light and event modalities, respectively. ⊙ represents pointwise multiplication, and expand(M) copies M along the channel dimension to... OpticalFlow(·) represents an optical flow network, which consists of linear layers and is used to extract motion information from frames. λ is the motion weight coefficient.

6. The moving target detection method based on visible light-event fusion according to claim 1, characterized in that, For f r and f e The process involves feature fusion and target detection based on the fused features. The final detection result includes the following: f e and f r Perform concatenation and convolution operations to generate multi-scale fused features f': f'=Cat(f e ,f r ) Where Cat(·) represents feature concatenation; Inputting f' into the feedforward neural network outputs the prediction result: Output = FFN(f') Where FFN(·) represents a feedforward neural network, and Output represents the list of predicted candidate boxes output.

7. A visible light-event fusion moving target detection system, based on the visible light-event fusion moving target detection method according to any one of claims 1-6, characterized in that, include: The module includes a data processing module, a random initialization object query module, a potential moving object sensor, a feature extraction module, and a spatial perception feature localization module. The data processing module is used to select historical datasets including visible light-event image pairs and to preprocess the historical datasets. The random initialization object query module is used to randomly initialize visible light and time object queries separately, and obtain the visible light object query q. rgb And query the event object q event ; A latent moving object perceptron, used to update the initial object query, includes a multimodal large language model, a cross-attention unit, and a feature mapping unit, wherein: Multimodal large language model for use with visible light image X RGB The prior position information F of the potential moving object is obtained from the input of the prompt text. pos ; Cross-attention units are used to process prior location information F pos Visible light object lookup using cross-attention CA(·) rgb And query the event object q event The initialization yields q' respectively. rgb and q' event ; Feature mapping unit, used to map q' rgb and q' event The final visible light object query is obtained after feature mapping. and final event object query Use the final visible light object query Update q rgb Use the final event object to query Update q event ; The feature extraction module includes a visible light feature extraction branch network and an event feature extraction branch network, which are used to extract features from visible light images X, respectively. RGB and event image X EVENT Extract visible light features f rgb and event characteristics f event ; The spatially aware feature localization module includes a text decoder and a spatial weighting unit, wherein: A text decoder is used to process the prior location information F output by a multimodal large language model. pos Decode the data to obtain the normalized coordinates B corresponding to the prior location information; Spatial weighting cells are used to obtain the visible light characteristics f of the final target region after spatial weighting based on normalized coordinates B. r and the event characteristics of the final target area f e ; The feature fusion module is used to combine f r f e , and The data is sent to the decoder, and the object is queried simultaneously. and The coordinates of the reference point of the potential moving object are obtained by linear projection. The decoder extracts the object feature f from the feature map based on the reference point coordinates. r and f e and complete the task for f r and f e Feature fusion is performed, and target detection is carried out based on the fused features to obtain the final detection result.

Images