A night farmland slender obstacle detection method and system based on multi-modal fusion and dynamic attention mechanism

Abstract

The application discloses a kind of night farmland slender obstacle detection method and system based on multi-modal fusion and dynamic attention mechanism, method includes the multi-modal data of the acquisition to be detected target, the multi-modal data includes visible light image, infrared image and millimeter wave radar point cloud data;The multi-modal data collected is preprocessed, and the data after processing is obtained;The preprocessing includes time synchronization, space alignment and data enhancement;Detection model is constructed, and the detection model includes multi-modal fusion module and dynamic attention module;The detection model is trained, and the model after training is obtained;Using the model after training detects the target to be detected;The application can improve detection precision and accuracy in small obstacle target detection, and enhance robustness in strong light or dust environment, improve the safety of plant protection unmanned aerial vehicle operation.

Description

Technical Field

[0001] This invention belongs to the field of intelligent obstacle avoidance technology for agricultural drones, and in particular relates to a method and system for detecting slender obstacles in farmland at night based on multimodal fusion and dynamic attention mechanism. Background Technology

[0002] Current nighttime farmland obstacle detection technologies have the following shortcomings: pure vision solutions are prone to missing small obstacles in low light conditions, while lidar has insufficient accuracy in recognizing slender linear structures (such as wires), and single sensors have limitations; traditional algorithms have a high false detection rate in strong light interference, dusty or rainy / foggy weather, and cannot adjust detection strategies in real time, resulting in poor adaptability to dynamic environments. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide a method and system for detecting slender obstacles in farmland at night based on multimodal fusion and dynamic attention mechanisms, which can improve detection accuracy.

[0004] Technical solution: The present invention provides a method for detecting slender obstacles in farmland at night based on multimodal fusion and dynamic attention mechanism, comprising:

[0005] S1: Collect multimodal data of the target to be detected, including visible light images, infrared images and millimeter-wave radar point cloud data;

[0006] S2: Preprocess the collected multimodal data to obtain processed data; the preprocessing includes time synchronization, spatial alignment and data augmentation;

[0007] S3: Construct a detection model, which includes a multimodal fusion module and a dynamic attention module;

[0008] S4: Train the detection model to obtain the trained model;

[0009] S5: Use the trained model to detect the target.

[0010] Furthermore, the multimodal fusion module described in step S3 includes a feature extraction architecture and a feature fusion architecture.

[0011] Furthermore, the feature extraction architecture uses a lightweight convolutional neural network (CNN) to extract features from visible light and infrared images. The lightweight CNN adds a lightweight edge enhancement branch to the output layer to strengthen edge features and outputs feature maps at three scales: 80×80, 40×40, and 20×20.

[0012] The PointNett network framework is used to process the features of millimeter-wave radar point cloud data. The PointNett network framework is divided into a sampling layer, a grouping layer and a feature propagation layer, which are used to extract local and global features of millimeter-wave radar point cloud data, and attention pooling is introduced in the feature extraction process.

[0013] Furthermore, the lightweight edge enhancement branch has the following specific structure: a Convolution is used for channel reduction; the result of the reduction is then processed using... Depthwise separable convolutions are used to extract features and reduce the number of parameters. These depthwise separable convolutions include depthwise convolutions and pointwise convolutions. The extracted features are then processed using a fixed Sobel operator to extract gradients in the horizontal and vertical directions and calculate the gradient magnitudes. The gradient magnitude features are then adjusted using a learnable 1x1 convolution.

[0014] Furthermore, the feature fusion architecture includes three layers: an early fusion layer, a mid-term fusion layer, and a late-term fusion layer.

[0015] The early fusion specifically involves stitching the aligned visible light image and infrared image together by channel at the data layer, and inputting them into the first convolutional layer of a lightweight convolutional neural network (CNN) to achieve the fusion of underlying texture features.

[0016] The intermediate fusion specifically involves combining element-wise addition and channel concatenation with the aligned millimeter-wave radar point cloud feature maps at the three scale output layers of the lightweight convolutional neural network (CNN) to achieve fusion.

[0017] The late-stage fusion specifically involves performing independent target detection on the fusion branch of visible light image and infrared image and the fusion branch of millimeter-wave radar point cloud data. For each branch, the detected target category, location, and confidence level are used as the preliminary detection results. The dynamic attention module outputs the weight of this branch, and the preliminary detection results of the two branches are fused using a weighted voting method.

[0018] Furthermore, the independent target detection includes:

[0019] For the feature map of the fusion branch of visible light and infrared images, an anchor-box-based detection head is adopted. A set of anchor boxes is predefined as candidate detection boxes. Then, the anchor box position, size and category are adjusted through neural network learning to match the target. First, the bounding box offset, target category score and target confidence are predicted at each feature map scale. Then, the category, position and confidence of each detected target are output.

[0020] On the BEV map output by the fusion branch of millimeter-wave radar point cloud data, a 2D target detection network is used to output the position, category, and confidence of the target in the BEV coordinate system. Finally, the radar branch outputs the category, position, and confidence of each detected target.

[0021] Furthermore, the dynamic attention module described in step S3 specifically includes:

[0022] Using light intensity, weather conditions, and risk level as environmental parameters, the environmental parameters are normalized to the [0,1] interval to construct an environmental feature vector. Based on a conditional attention network, the environmental feature vector is input and transformed through two fully connected layers, outputting weight coefficients for visible light images, infrared images, and millimeter-wave radar point cloud data, respectively. The Softmax function is used as the activation function, making the range of each weight coefficient [0,1]. An LSTM-based gating mechanism is introduced, using the weight coefficients output by the conditional attention network as gating parameters to perform weighted fusion of features from each modality during mid-stage fusion. At the same time, the weight coefficients are adjusted according to preset adaptive rules.

[0023] Based on the same inventive concept, this invention also provides a nighttime farmland slender obstacle detection system based on multimodal fusion and dynamic attention mechanisms, comprising:

[0024] The acquisition module is used to acquire multimodal data of the target to be detected, including visible light images, infrared images, and millimeter-wave radar point cloud data.

[0025] The preprocessing module is used to preprocess the acquired multimodal data to obtain processed data; the preprocessing includes time synchronization, spatial alignment, and data augmentation.

[0026] A construction module is used to build a detection model, which includes a multimodal fusion module and a dynamic attention module;

[0027] The training module is used to train the detection model to obtain the trained model;

[0028] The detection module is used to detect the target object using the trained model.

[0029] Based on the same inventive concept, the present invention also provides a computing device, comprising: one or more processors, one or more memories, and one or more programs, the programs being stored in the memory and configured to be executed by the processor, wherein when the programs are loaded onto the processor, they implement the steps of the nighttime farmland slender obstacle detection method based on multimodal fusion and dynamic attention mechanism as described in any of the preceding claims.

[0030] Based on the same inventive concept, the present invention also provides a storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the steps of the nighttime farmland slender obstacle detection method based on multimodal fusion and dynamic attention mechanism as described in any of the preceding claims.

[0031] Beneficial effects: Compared with existing technologies, this invention proposes a target detection method with multimodal fusion and dynamic attention enhancement, which realizes real-time output of information such as target category, location, distance, and speed. This provides reliable environmental perception support for mobile platforms such as drones and autonomous driving, meets the requirements of low latency and high reliability in edge computing scenarios, and can solve the problem of missed detection of slender obstacles in farmland at night. It can not only improve the detection accuracy and precision in the detection of small obstacles, but also enhance robustness in strong and weak light or dusty environments, thereby improving the safety of agricultural drone operations. Attached Figure Description

[0032] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0034] Example 1, as shown in the attached document Figure 1 As shown in this embodiment, the nighttime farmland slender obstacle detection method based on multimodal fusion and dynamic attention mechanism includes:

[0035] S1: Collect multimodal data of the target to be detected, including visible light images, infrared images and millimeter-wave radar point cloud data;

[0036] S2: Preprocess the collected multimodal data to obtain processed data; the preprocessing includes time synchronization, spatial alignment and data augmentation;

[0037] S3: Construct a detection model, which includes a multimodal fusion module and a dynamic attention module;

[0038] S4: Train the detection model to obtain the trained model;

[0039] S5: Use the trained model to detect the target.

[0040] Specifically, in step S1, multimodal data acquisition is employed, fusing visible light images, infrared images, and millimeter-wave radar point cloud data to compensate for the performance deficiencies of single-modal acquisition in extreme environments and achieve all-weather detection. During data acquisition, a unified signal triggering mechanism is used to synchronously control the acquisition of the three types of data, ensuring the consistency of the initial time of the raw data. At the same time, a timestamp calibration mechanism is introduced to correct for lost or interpolated data with deviations, avoiding the impact of asynchronous data on subsequent fusion results.

[0041] Visible light camera: captures RGB three-channel visible light images, records rich texture and color information in the environment, and is suitable for well-lit daytime scenes.

[0042] Infrared sensor: Collects infrared images and acquires single-channel thermal radiation images, enhancing the ability to identify slender obstacles (such as wires and tree branches) in nighttime, foggy, and low-light environments.

[0043] Millimeter-wave radar: Collects millimeter-wave radar point cloud data. Each point cloud map contains information such as distance, azimuth, elevation, and velocity, providing information such as distance, velocity, and azimuth of obstacles. It is suitable for all-weather detection.

[0044] In step S2, the original data is calibrated and aligned to output standardized data with a unified format and coordinates, providing high-quality input for feature extraction. The core process includes three steps: time synchronization, spatial alignment, and data augmentation.

[0045] Time Synchronization: Ensures timestamp alignment of multi-sensor data through hardware triggering or software synchronization. Hardware-triggered hard synchronization is the core, with the trigger signal period matching the camera frame rate. Radar point cloud acquisition and image acquisition trigger signals are synchronized, ensuring timestamp errors of the three types of data are ≤1ms. A timestamp verification queue is built in the software layer to compare the timestamps of each frame of data. If the deviation exceeds 1ms, a retry of acquisition is triggered; if deviations occur in three consecutive frames, a synchronization anomaly alarm is output, and historical data interpolation is enabled (for emergency use only).

[0046] Spatial Alignment: The core objective is to uniformly map infrared images and millimeter-wave radar point cloud data onto the pixel coordinate system of visible light images, achieving spatial registration of multimodal data. Camera Intrinsic Calibration: Obtain focal length, principal point, and distortion coefficients. Radar and Camera Extrinsic Calibration: Obtain the rotation and translation matrices between the two. Joint Calibration: Using a calibration board, simultaneously obtain the extrinsic parameters of both the camera and radar, projecting the millimeter-wave radar point cloud data onto the image coordinate system.

[0047] Data augmentation involves rotating, scaling, and adjusting brightness of images to improve model generalization and avoid overfitting. For visible light images and aligned infrared images, conventional augmentation (rotation, scaling, horizontal flipping, etc.), illumination-adaptive augmentation (brightness, contrast, etc.), and noise injection (Gaussian noise, etc.) are used. For millimeter-wave radar point cloud data, point cloud sampling, noise addition, and point cloud stitching are employed to enhance the image. During data augmentation, the spatiotemporal alignment of multimodal data must be maintained to ensure that the position of the same target remains consistent across different modalities after augmentation.

[0048] Step S3: Construct a detection model, which includes a multimodal fusion module and a dynamic attention module.

[0049] The multimodal fusion module adopts a two-level architecture of "feature extraction-feature fusion". It designs a dedicated feature extraction network for the characteristics of different modal data and achieves information complementarity through a multi-level fusion strategy to improve target detection performance.

[0050] Feature extraction includes: using lightweight convolutional neural networks (CNNs) (such as ResNet and YOLO8-backbone) to extract features from visible light and infrared images, balancing feature extraction capability with inference speed. For some slender obstacles, a lightweight edge enhancement branch can be added to the output layer to strengthen edge features. Three feature maps of different scales are used as outputs to apply feature extraction for small, medium, and large targets respectively.

[0051] PointNet or a lightweight radar network is used to process millimeter-wave radar point cloud data features, simplifying the sampling and grouping modules and improving inference speed. The network consists of a sampling layer (FPS sampling), a grouping layer (sphere query grouping), and a feature propagation layer (interpolation fusion) to extract local and global features of the point cloud. Attention pooling is introduced during feature extraction to strengthen the feature weights of the target point cloud and suppress interference from the background point cloud.

[0052] The addition of a lightweight edge enhancement branch aims to highlight edge information in the image, especially for slender obstacles (such as wires and tree branches) and obstacle contours, in order to improve detection accuracy.

[0053] This lightweight edge enhancement branch enhances edges without affecting real-time performance by using gradient or high-frequency information. It then fuses the enhanced features with the original features, allowing the network to focus on both texture and edge information simultaneously.

[0054] The edge enhancement branch consists of the following parts:

[0055] (1) One Convolution reduces the dimensionality of channels, thus reducing computational load.

[0056] (2) Use Depthwise separable convolutions further extract features while reducing the number of parameters. Depthwise separable convolutions include depthwise convolution (channel-wise convolution) and pointwise convolution (…). convolution).

[0057] (3) Edge enhancement module: Use two fixed Sobel operators (horizontal and vertical) or learnable edge detection operators (e.g. Convolution is used to extract edge features. To reduce scalability, a fixed Sobel operator is used to extract gradients in the horizontal and vertical directions, and then the gradient magnitudes are calculated.

[0058] (4) Transform the gradient magnitude features through a learnable... The convolution is adjusted and then fused with the original features.

[0059] Feature fusion includes a multi-layered fusion architecture of "early fusion + mid-term fusion + late-term fusion" at the feature level, which makes full use of feature information at different levels to achieve deep complementarity of multimodal information.

[0060] Early fusion: In the data layer, the aligned visible light image and infrared image are stitched together by channel and input into the first convolutional layer of the lightweight convolutional neural network (CNN) to achieve the fusion of low-level texture features.

[0061] Mid-term fusion: In the three scale output layers of the lightweight convolutional neural network (CNN), a fusion method is used that combines element-wise addition (fusion of global information) and channel concatenation (preservation of detailed information) with the aligned millimeter-wave radar point cloud feature maps.

[0062] Late-stage fusion: Independent target detection is performed on the visible light image and infrared image fusion branch and the millimeter-wave radar point cloud data fusion branch to obtain preliminary detection results (target category, location, confidence level). The detection results of the two branches are then fused using a weighted voting method, with the weights output by the dynamic attention module.

[0063] Specifically, for the feature map output by the fusion branch of visible light image and infrared image, an anchor-based detection head is adopted. That is, a set of anchor boxes are predefined as candidate detection boxes, and then the position, size and category of the anchor boxes are adjusted by the neural network to match the real target.

[0064] First, predict the bounding box offset, object class score, and object confidence (whether the object is contained) at each feature map scale. Then, output the following for each detected object: class (vehicle, pedestrian, power line, etc.), bounding box (position in the image coordinate system), and confidence score.

[0065] On the BEV map output by the fusion branch of millimeter-wave radar point cloud data, a 2D target detection network is used to output the target's position (x, y, z, length, width, height, orientation angle) in the BEV coordinate system, as well as its category and confidence score. Finally, the radar branch outputs for each detected target: category (vehicle, pedestrian, etc.; note that radar may not be able to detect slender obstacles such as power lines), 3D bounding box (position and size in the radar coordinate system), and confidence score.

[0066] After the two branches perform target detection independently, their detection results are fused. Since the detection results from the two branches are in different coordinate systems (image coordinate system and radar coordinate system), they need to be transformed to the same coordinate system (such as world coordinate system or vehicle coordinate system). The fusion strategy is as follows:

[0067] 1. Project the 3D bounding box detected by the radar onto the image plane to obtain the 2D bounding box.

[0068] 2. For each target detected by the visible light image and infrared image fusion branch, calculate its IoU (Intersection over Union, which measures the degree of overlap between two regions) with the 2D bounding box of the radar projection.

[0069] 3. If the IoU is greater than a threshold (e.g., 0.5), the two detections are considered to be the same target, and then the following fusion is performed:

[0070] (1) Category: The category is based on the fusion branch of visible light image and infrared image (because image classification is more accurate).

[0071] (2) Location: The 3D location is based on the millimeter-wave radar point cloud data fusion branch (because radar ranging is more accurate).

[0072] (3) Confidence: Take the weighted average of the confidence of the two branches (the weights can be adjusted, such as 0.7 for radar and 0.3 for image).

[0073] For targets (such as power lines) detected by the visible light and infrared image fusion branch but not by the millimeter-wave radar point cloud data fusion branch, the detection results of the visible light and infrared image branches are retained, but their 3D positions are set to estimated values.

[0074] For targets detected by the millimeter-wave radar point cloud data fusion branch but not detected by the fusion of visible light and infrared images (such as obscured vehicles), the detection results of the millimeter-wave radar point cloud data fusion branch are retained, but the category may be inaccurate (it can be marked as unknown or classified according to radar characteristics).

[0075] The dynamic attention module, by introducing environmental perception and scene risk assessment, dynamically adjusts the fusion weights of multimodal features to achieve "scene-adaptive" target detection. The core design concept is "environmental input - risk assessment - weight allocation - fusion control".

[0076] Environmental Input: Key environmental parameters affecting multimodal detection performance are collected to provide data support for attention weight adjustment. Information such as light intensity, weather conditions, and risk level are collected, and environmental parameters are normalized to the [0,1] interval to construct an environmental feature vector, which is then input into the attention network.

[0077] Dynamic weight allocation mechanism: A dual control mechanism of "conditional attention network + gated fusion" is adopted to dynamically adjust the weights of each modality feature based on the environmental feature vector. Specifically, after inputting the environmental feature vector, feature transformation is performed through two fully connected layers, outputting three weight coefficients for visible light, infrared, and radar respectively. The activation function is the Softmax function, ensuring that the coefficient range is [0,1]. In the intermediate fusion, a gate unit (based on LSTM gating mechanism) is introduced, using the weight coefficients output by the conditional attention network as gating parameters to perform weighted fusion of image features and radar features. Scene adaptive rules are set, that is, three weight coefficient values ​​are set for light intensity, weather, risk level, etc.

[0078] In step S4, focusing on the goals of "high precision + lightweight," a scientific training strategy and model optimization scheme are designed to ensure a balance between model performance and efficiency on edge devices. This includes:

[0079] S41: Dataset Construction: Collect and label multimodal data (visible light images, infrared images, and millimeter-wave radar point cloud data), labeling target categories and locations. Data collection should include various special scenarios, such as slender obstacles and regular target samples. Data labeling should adopt the COCO dataset format standard, including target categories (8 categories: vehicles, pedestrians, cyclists, power lines, tree branches, guardrails, signs, and other obstacles), bounding boxes (image modalities), and 3D coordinates (radar modalities). Simultaneously, the ratio of training set, validation set, and test set samples should be 7:2:1, ensuring uniform distribution of samples across environmental conditions and target categories during partitioning.

[0080] S42: Loss function design: Combine classification loss (such as Cross-Entropy) and localization loss (such as IoU Loss) to solve the class imbalance problem caused by the scarcity of samples of slender obstacles, and improve the accuracy of target localization, especially the accuracy of bounding box regression of slender obstacles.

[0081] S43: Training Strategy: Pre-train on single-modal data first, then perform multi-modal fusion training. First, feature extraction is performed on visible light images, infrared images, and millimeter-wave radar point cloud data separately. Transfer learning is used to initialize weights, such as using pre-trained weights from YOLOv8-backbone on the COCO dataset and loading pre-trained weights from ModelNet40 using PointNet++ to accelerate model convergence. Then, the pre-trained modules are combined, trained, and the fusion module and attention parameters are optimized.

[0082] S44: Use lightweight networks (such as MobileNet and ShuffleNet) to adapt to edge computing devices. While ensuring verification accuracy, use techniques such as quantization and pruning to compress the model size and improve inference speed.

[0083] This embodiment also includes deployment and real-time inference: the trained lightweight model is deployed on edge devices such as Jetson, NVIDIA TAO, or Huawei Atlas, enabling real-time reading, fusion, and inference of multi-sensor data.

[0084] The inference process includes: multi-threaded parallel reading of data from three sensors; performing preprocessing and alignment, including time synchronization, spatial alignment, and data augmentation; inputting the data into the fusion model for target detection; and outputting information such as target category, location, distance, and velocity.

[0085] Step S5 also includes detection result output and control: the detection results are encapsulated into a structure or JSON format, including: target type (vehicle, pedestrian, power line, etc.), location (bounding box, distance), confidence level, risk level, and output to the UAV control system. The detection results are transmitted via UART, CAN, or Ethernet interface, and the system makes decisions such as obstacle avoidance, path replanning, or hovering based on the results.

[0086] Based on the same inventive concept, this embodiment also provides a nighttime farmland slender obstacle detection system based on multimodal fusion and dynamic attention mechanism, including:

[0087] The acquisition module is used to acquire multimodal data of the target to be detected, including visible light images, infrared images, and millimeter-wave radar point cloud data.

[0088] The preprocessing module is used to preprocess the acquired multimodal data to obtain processed data; the preprocessing includes time synchronization, spatial alignment, and data augmentation.

[0089] A construction module is used to build a detection model, which includes a multimodal fusion module and a dynamic attention module;

[0090] The training module is used to train the detection model to obtain the trained model;

[0091] The detection module is used to detect the target object using the trained model.

[0092] Based on the same inventive concept, this embodiment also provides a computing device, including: one or more processors, one or more memories, and one or more programs, the programs being stored in the memory and configured to be executed by the processor, the programs being loaded onto the processor to implement the steps of the nighttime farmland slender obstacle detection method based on multimodal fusion and dynamic attention mechanism according to any one of the preceding claims.

[0093] Based on the same inventive concept, this embodiment also provides a storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the steps of the nighttime farmland slender obstacle detection method based on multimodal fusion and dynamic attention mechanism according to any one of the preceding claims.

Claims

1. A night farmland slender obstacle detection method based on multi-modal fusion and dynamic attention mechanism, characterized in that, include: S1: Collect multimodal data of the target to be detected, including visible light images, infrared images and millimeter-wave radar point cloud data; S2: Preprocess the collected multimodal data to obtain processed data; the preprocessing includes time synchronization, spatial alignment and data augmentation; S3: Construct a detection model, which includes a multimodal fusion module and a dynamic attention module; S4: Train the detection model to obtain the trained model; S5: Use the trained model to detect the target.

2. The method for detecting slender obstacles in farmland at night based on multimodal fusion and dynamic attention mechanism according to claim 1, characterized in that, The multimodal fusion module mentioned in step S3, This includes feature extraction architecture and feature fusion architecture.

3. The method for detecting slender obstacles in farmland at night based on multimodal fusion and dynamic attention mechanism according to claim 2, characterized in that, The feature extraction architecture uses a lightweight convolutional neural network (CNN) to extract features from visible light and infrared images. The lightweight CNN adds a lightweight edge enhancement branch to the output layer to strengthen edge features and outputs feature maps at three scales: 80×80, 40×40, and 20×20. The PointNett network framework is used to process the features of millimeter-wave radar point cloud data. The PointNett network framework is divided into a sampling layer, a grouping layer and a feature propagation layer, which are used to extract local and global features of millimeter-wave radar point cloud data, and attention pooling is introduced in the feature extraction process.

4. The method for detecting slender obstacles in farmland at night based on multimodal fusion and dynamic attention mechanism according to claim 3, characterized in that, The lightweight edge enhancement branch has the following structure: a Convolution is used for channel reduction; the result of the reduction is then processed using... Depthwise separable convolution extracts features and reduces the number of parameters. This depthwise separable convolution includes depthwise convolution and pointwise convolution. The extracted features are then processed using a fixed Sobel operator to extract horizontal and vertical gradients, and the gradient magnitudes are calculated. These gradient magnitude features are then processed through a learnable... The convolution is adjusted.

5. The method for detecting slender obstacles in farmland at night based on multimodal fusion and dynamic attention mechanism according to claim 3, characterized in that, The feature fusion architecture includes three layers: an early fusion layer, a mid-term fusion layer, and a late-term fusion layer. The early fusion specifically involves stitching the aligned visible light image and infrared image together by channel at the data layer, and inputting them into the first convolutional layer of a lightweight convolutional neural network (CNN) to achieve the fusion of underlying texture features. The intermediate fusion specifically involves combining element-wise addition and channel concatenation with the aligned millimeter-wave radar point cloud feature maps at the three scale output layers of the lightweight convolutional neural network (CNN) to achieve fusion. The late-stage fusion specifically involves performing independent target detection on the fusion branch of visible light image and infrared image and the fusion branch of millimeter-wave radar point cloud data. For each branch, the detected target category, location, and confidence level are used as the preliminary detection results. The dynamic attention module outputs the weight of this branch, and the preliminary detection results of the two branches are fused using a weighted voting method.

6. The method for detecting slender obstacles in farmland at night based on multimodal fusion and dynamic attention mechanism according to claim 5, characterized in that, The independent target detection includes: For the feature map of the fusion branch of visible light and infrared images, an anchor-box-based detection head is adopted. A set of anchor boxes is predefined as candidate detection boxes. Then, the anchor box position, size and category are adjusted through neural network learning to match the target. First, the bounding box offset, target category score and target confidence are predicted at each feature map scale. Then, the category, position and confidence of each detected target are output. On the BEV map output by the fusion branch of millimeter-wave radar point cloud data, a 2D target detection network is used to output the position, category, and confidence of the target in the BEV coordinate system. Finally, the radar branch outputs the category, position, and confidence of each detected target.

7. The method for detecting slender obstacles in farmland at night based on multimodal fusion and dynamic attention mechanism according to claim 6, characterized in that, The dynamic attention module mentioned in step S3 specifically includes: Using light intensity, weather conditions, and risk level as environmental parameters, the environmental parameters are normalized to the [0,1] interval to construct an environmental feature vector. Based on a conditional attention network, the environmental feature vector is input and transformed through two fully connected layers, outputting weight coefficients for visible light images, infrared images, and millimeter-wave radar point cloud data, respectively. The Softmax function is used as the activation function, making the range of each weight coefficient [0,1]. An LSTM-based gating mechanism is introduced, using the weight coefficients output by the conditional attention network as gating parameters to perform weighted fusion of features from each modality during mid-stage fusion. At the same time, the weight coefficients are adjusted according to preset adaptive rules.

8. A nighttime farmland slender obstacle detection system based on multimodal fusion and dynamic attention mechanism, characterized in that, include: The acquisition module is used to acquire multimodal data of the target to be detected, including visible light images, infrared images, and millimeter-wave radar point cloud data. The preprocessing module is used to preprocess the collected multimodal data to obtain the processed data. The preprocessing includes time synchronization, spatial alignment, and data augmentation; A construction module is used to build a detection model, which includes a multimodal fusion module and a dynamic attention module; The training module is used to train the detection model to obtain the trained model; The detection module is used to detect the target object using the trained model.

9. A computing device, characterized in that, include: One or more processors, one or more memories, and one or more programs, said programs being stored in the memory and configured to be executed by the processor, said programs being loaded onto the processor to implement the steps of the nighttime farmland slender obstacle detection method based on multimodal fusion and dynamic attention mechanism according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the steps of the nighttime farmland slender obstacle detection method based on multimodal fusion and dynamic attention mechanism according to any one of claims 1 to 7.

Images