A multi-modal environment perception system for autonomous driving and a method thereof

Abstract

This invention discloses a multimodal environment perception system and method for autonomous driving, belonging to the field of intelligent connected vehicle technology. The system includes an in-vehicle imaging subsystem, an AMSR-T image enhancement module, and a 77GHz millimeter-wave radar subsystem. It achieves environmental perception through multimodal data acquisition, image enhancement, target extraction and tracking, BEV transformation, cross-modal fusion, and semantic decision-making. The method involves data acquisition, image enhancement, target processing, coordinate projection, feature fusion, and result output. The multimodal environment perception system and method provided by this invention solve the shortcomings of single-modal perception in low-light, rain, and fog scenarios, as well as the high cost and latency issues of existing fusion schemes. It achieves end-to-end low latency and low power consumption through a heterogeneous computing platform, making it suitable for autonomous driving environment perception.

Description

Technical Field

[0001] This invention relates to the field of intelligent connected vehicle technology, and in particular to a multimodal environmental perception system and method for autonomous driving. Background Technology

[0002] Current technologies still have the following shortcomings:

[0003] (1) Single-modal sensing defects

[0004] In nighttime and rain / fog scenarios, the quantum efficiency of CMOS sensors drops sharply; the rolling-shutter effect, combined with vehicle vibration, causes stripe misalignment, resulting in a more than 30% decrease in mAP detected by the classic YOLO / DETR series. Millimeter-wave radar is anti-obstruction and all-weather, but the angular resolution of 4TX-8RXMIMO at 77GHz is limited (typically 1.6°), and when used alone, it suffers from target shape blurring and cluster fusion problems.

[0005] (2) Existing integration solutions are insufficient

[0006] One type of method (such as BEVFusion) relies on 32-128 line LiDAR, which is not suitable for cost-sensitive L2+ / L3 mass production; the second type of method (such as 100Hz-level frame-level stitching) ignores timing drift and is difficult to complete complete inference within 25ms automotive-grade latency.

[0007] (3) The dual challenges of low light and shake

[0008] The camera operates in low light (<2 lux) and with shaking (>50° / s). 2 In complex environments, objects simultaneously suffer from photon noise, motion blur, and exposure drift. Current Retinex or Deblur networks, when used alone, cannot simultaneously achieve noise suppression and texture fidelity.

[0009] Therefore, there is an urgent need for an end-to-end, low-power, mass-producible overall solution that combines "image enhancement + radar supplementation + spatiotemporal alignment + lightweight fusion". Summary of the Invention

[0010] The purpose of this invention is to provide a multimodal environment perception system and method for autonomous driving, which solves the problem of [missing information].

[0011] To achieve the above objectives, the present invention provides a multimodal environment perception system for autonomous driving, comprising:

[0012] The vehicle-mounted imaging subsystem is used to acquire raw image sequences I s ;

[0013] The Adaptive Multi-Scale Retinex-Transformer Image Enhancement Module (AMSR-T) takes I as input.s And output enhanced image

[0014] The 77GHz millimeter-wave radar subsystem is used to output time-range-Doppler three-dimensional cube C for inter-frame registration. t (r,f d ,θ);

[0015] The radar target extraction and tracking module, for C t The target list R is obtained through processing. s The bird's-eye view (BEV) transformation module will... With R s All projections are uniformly directed to the same BEV reference frame;

[0016] The cross-modal Transformer-GAT fusion network performs graph attention fusion on BEV image raster nodes and radar voxel nodes, outputting joint features F. s ;

[0017] The semantic decision module, based on F s Output target category, location, and trajectory information;

[0018] The heterogeneous computing platform consists of Jetson OrinNX (responsible for deep learning inference) and Zynq Ultra Scale+ MPSoC (responsible for radar signal preprocessing), with an end-to-end inference latency of less than 25ms and a total system power consumption of no more than 6W.

[0019] Preferably, the AMSR-T module performs multi-scale Retinex luminance recovery using the following formula:

[0020]

[0021] in: The representative scale is σ s The smoothing kernel of a Gaussian filter is used to blur images. s This is the standard deviation of the Gaussian kernel, used to control the degree of blur. The weight ω... s Dynamically generated by the Transformer self-attention mechanism, the weights assigned to each layer during the fusion process reflect the influence of each layer and satisfy the following conditions: “Clip” indicates that the result is saturated and cropped to suppress noise, which is used to limit the intensity value of the enhanced image to a certain range (such as [0,1]). The enhanced image refers to the final output image after image enhancement (such as Retinex enhancement). sThe s-th layer image of the input image is typically the image smoothed by different Gaussian filters. Equation (1) describes the image processing using multi-scale Retinex enhancement. First, Gaussian filtering is applied to blur the input image, then brightness restoration is performed, and finally, images at different scales are weighted and fused to generate the final enhanced image.

[0022] Preferably, the AMSR-T module includes a deblurring branch based on gyroscope inertial measurement unit (IMU) angular velocity estimation, and the predicted fuzzy kernel formula is as follows:

[0023]

[0024] In equation (2), k t The predicted blur kernel is dynamically adjusted based on the gyroscope's angular velocity to remove motion blur from the image. Here, u and v are the kernel coordinates, and θ... t The cumulative rotation angle (in radians) within the exposure window of that frame. Equation (2) expresses the variable dependencies of the PSF (point spread function) generated by the inertial measurement unit (IMU). The angular velocity ω(τ) output by the IMU sensor is used to estimate the motion of the vehicle or camera, thereby calculating the size of the blur kernel. t The kernel reflects the degree of motion blur in the image and is used in subsequent deblurring processes.

[0025] Its convolution kernel size satisfies:

[0026]

[0027] Where β is the empirical proportionality coefficient, and ω(τ) is the IMU three-axis angular velocity vector at time τ (generally in rad / s), which usually represents the physical quantity of an object in a gyroscope, rotating platform, or any sensor that needs to measure the rotation rate when it is in motion such as rotation, yaw, or pitch. The integral of the angular velocity over time from time t-Δt to t is the change in angle (in rad) during that time. In the discretization implementation, if the sampling rate is very high, it can be approximated as...

[0028]

[0029] Where ω[k] is the angular velocity component of the kth sampling point, and Δt is the exposure time window. |·| Absolute value operation is used to extract non-negative "amplitude" information. For example, if the integration result is positive / negative, the absolute value is positive, indicating that regardless of the positive / negative direction, only the "magnitude of rotation" matters. Δt is the time interval. β is the scaling factor (scalar), used to scale the original absolute value result... The value can be scaled up or down to a suitable range as needed. The specific value is usually determined by the system designer based on dimensions, range, or experience. d is the final output scalar, representing the "angular velocity integral amplitude" or "rotation index" after scaling. Equation (3) indicates that d can be used to trigger threshold judgment, adjust control strategy, update memory unit, etc. in different application scenarios. For example, when the rotation amplitude exceeds a certain threshold, it is considered that a significant rotation has occurred, which in turn affects the subsequent process.

[0030] Preferably, the radar target extraction and tracking module employs an improved spatiotemporal constant false alarm rate (ST-CFAR) algorithm with a threshold of [missing information].

[0031] T(r,f d )=μ(r,f d )+ασ(r,f d (4);

[0032] It also combines Kalman-LSTM filtering to achieve multi-frame target association and trajectory smoothing. T(r,f) d The threshold value of the radar signal is used for target extraction and filtering. μ(r,f) d The mean value of the radar signal (usually the mean value of the background noise in the signal area). α is a constant representing the gain coefficient of the threshold. σ(r,f) d The standard deviation of the radar signal represents the signal's volatility or noise amplitude. Equation (4) defines the adaptive threshold T(r,f) of the radar signal. d The calculation is based on the mean and standard deviation of the radar signal. By setting an appropriate gain coefficient α, the target can be effectively distinguished from noise.

[0033] Preferably, the BEV transformation module uses a differentiable homography matrix:

[0034] H cam→BEV =K[r1 r2 t](5);

[0035] Assuming the ground is a plane Z=0 and the angle between the camera's Z-axis and the ground normal vector is already represented in r1, r2, t, then the pixels can be projected onto the BEV mesh. The coordinate system has been normalized H. cam→BEV The homography matrix from the camera coordinate system to the bird's-eye view (BEV) coordinate system. The intrinsic parameter matrix of camera K describes the camera's focal length, principal point, and other parameters. The first two columns of the rotation matrices of cameras r1 and r2 represent the camera's direction vectors on the three coordinate axes, respectively. The translation vector of camera t describes the camera's position in three-dimensional space. Equation (5) describes how to map the image from the camera coordinate system to the bird's-eye view (BEV) coordinate system using the camera's intrinsic and extrinsic parameters. Homography matrix H cam→BEV This is used to project image features from the camera's viewpoint onto a top-down view, facilitating subsequent cross-modal fusion. The image features F...img Project onto BEV grid coordinates (u,v) and fill the holes using a self-attention-based density mask.

[0036] Preferably, the node attention weights of the cross-modal Transformer-GAT network satisfy:

[0037]

[0038] in The cosine similarity function reflects the calculation of the query vector q based on cosine similarity. i and key vector k j The degree of matching. T is a temperature coefficient used to smooth the calculation of attention weights. α ij Attention weights between node i and node j are used for cross-modal image and radar voxel matching. q i , represent the query vector and key vector of node i and node j, respectively, derived from image and radar data. Equation (6) is the standard formula for calculating attention weights in graph attention mechanisms. The similarity between the query vector and key vector is calculated using cosine similarity, and its value is adjusted using a temperature coefficient T. This calculation is used for cross-modal feature alignment and fusion.

[0039] Preferably, when the ambient illuminance L < 2 lux, the AMSR-T module automatically applies a gain g. L To improve contrast in low-light scenes, its expression is:

[0040] g L =1+0.3(2-L)(7).

[0041] Preferably, the system employs a time-gated fusion strategy:

[0042]

[0043] The left side of equation (8) is a weighted fusion or gating mechanism. This represents the feature or measurement value obtained directly from the sensor or a certain computing module at the current time t. This represents the features retained in the memory cell from the previous moment. γ t It is a scalar (gate value) ranging from [0,1], used to measure the fusion ratio of "current features" and "historical memories". If γ t If the value approaches 1, then the current feature is more trusted. If γ t A value closer to 0 indicates a greater inclination to preserve historical memory. The right side of equation (8) first concatenates or merges the two inputs Δv (change in velocity) and Δa (change in acceleration) and feeds them into a multi-layer perceptron (MLP). The output of the MLP then passes through a σ (Sigmoid) function σ(·) to ensure that the final γ t ∈[0,1]. The purpose of this is to allow the network to automatically "learn" how to assign fusion weights to current features and historical memories under different speed / acceleration changes.

[0044] Preferably, the target detection loss function is defined as follows:

[0045]

[0046] in The total loss function, used to optimize the object detection task. λ cls The weight of the classification loss controls the impact of the target category classification loss on the total loss. Focusing loss is used to address class imbalance problems and is typically used to handle difficult samples. λ loc The weight of position loss controls the impact on target positioning accuracy. Loss, used to optimize the accuracy of the target position. λ rad The weights of the radar-visual consistency loss control the consistency between radar and visual data. The registration loss between radar and vision is used to ensure spatial consistency in their fusion. Equation (9) represents that the loss function consists of multiple parts, including classification loss, location loss, IoU loss, and radar-vision consistency loss. The loss of each part is optimized by weighting to improve the detection accuracy and cross-modal consistency of the model.

[0047] Preferably, synthetic data augmentation based on a light-aerosol scattering model is used for rain and fog conditions, and cross-domain robust training is performed through an adversarial domain adapter D.

[0048] Preferably, the system provides a real-time visualization interface for outputting enhanced images, radar point clouds, BEV occupancy maps, and attention heatmaps for online diagnosis and parameter tuning.

[0049] Preferably, the system supports OTA (Over-the-Air) upgrades, and the Transformer-GAT weight blocks are replaced online after 4-bit and 8-bit layered quantization to keep the accuracy decrease by no more than 1%.

[0050] Preferably, the fusion result, combined with a high-precision map (HDMap), uses Bayesian prior constraints to constrain vehicle driving lanes and correct for lateral offset errors.

[0051] Preferably, a hybrid loss is used during the training phase:

[0052]

[0053] Where p sensory For a single-mode prediction distribution, p fusion The distribution is fused for prediction. η (KL term weight) can be initially set to around 0.1. During validation, observe the trade-off between the performance improvement of the single-modal branch in low-light scenes and the performance degradation after fusion. If the single-modal branch is still too weak, η can be increased appropriately; if the accuracy decreases significantly after fusion, η can be decreased. ξ (TripletLoss weight) can be initially set between 0.05 and 0.2. If a significant improvement is observed in target tracking or ID preservation, it can be increased appropriately; if the detection accuracy regresses, it can be decreased appropriately. Marginα (boundary in triplet loss) is adjusted within the range of [0.1, 1.0]. If the boundary is too large, negative samples will have difficulty satisfying the distance constraint, and the Loss will not converge; if the boundary is too small, the discrimination will be insufficient. The KL divergence temperature τ is consistent with the Softmax temperature mechanism of Transformer, and is generally chosen to be τ∈[0.05, 0.2]. The higher the temperature, the smoother the attention distribution; the lower the temperature, the "sharper" the attention distribution. Suppose we have the following three sample scenarios: In a low-light nighttime scene, the single-camera (vision) branch will output detection results with low confidence (e.g., due to noise interference). In this case, the multimodal fusion branch (which fuses radar information) will output a more robust prediction. Through the KL term, the vision branch gradually learns the "reliable" output distribution of the fused network. In a daytime open scene, the vision branch itself is powerful enough, and its output is similar to that of the fusion branch, with a small KL divergence. TripletLoss strengthens the clustering of the same target, reducing false detections of adjacent vehicles. In a rain / fog obscured scene, the single radar branch is prone to multipath noise at certain angles, while the fused distribution eliminates outlier information. KL loss makes the radar branch also move towards a more rational fusion distribution, and TripletLoss ensures that the feature space distance of the "same pedestrian" in different frames does not increase under rain / fog conditions. In all these scenarios, It integrates constraints from three aspects: "main task error", "cross-modal consistency" and "metric learning", thereby effectively improving the model's perceptual robustness and subsequent tracking performance under "low light jitter, occlusion and complex scenes". Common detection / location losses Cross-modal output consistency regularization (KL(·∥·)) and metric learning Organic combination. First item. The basic detection / localization loss represents the network's error in core tasks such as multimodal object detection, category classification, and bounding box regression. It includes the classification and localization losses of the vision branch and the radar branch respectively, and may also include the alignment loss between them (e.g., radar-vision consistency). For autonomous driving perception Minimizing this directly improves detection accuracy (mAP) and positioning accuracy (IoU). The second term ηKL(p sensory ||p fusion Cross-modal consistency constraints aim to make the prediction distribution of a single modality (e.g., vision only or radar only) approximate the more reliable prediction distribution after multimodal fusion as closely as possible. By introducing KL divergence, the model is encouraged to align the distribution of the results learned by the single-modal branch with that of the fused branch, thereby avoiding "collision" between the direction of the single sensor output and the fused result; the single-modal network itself is gradually "educated" by the strong robust features of the fused result, making the single sensor more stable in low-light, occluded, and other scenarios. The metric is if p sensory With p fusion The greater the difference, the higher the KL value, requiring a larger penalty; if the two are highly consistent, this item approaches 0. (Third item) Triplet metric learning aims to further narrow the embedding representations of different observations (visual frames, radar frames, different times) of the same physical target in the feature embedding space (i.e., the joint features output by Transformer-GAT), while simultaneously widening the embeddings of different physical targets. This enhances the clustering of similar targets (the same vehicle / pedestrian) in the feature space, improving tracking and trajectory consistency; and reduces confusion between different targets, minimizing false detections and intra-class drift.

[0054] Equation (10) is used in a typical application scenario. In a complex and congested environment, "Car A" has the same ID across multiple frames and should be kept relatively close; "Car A" and "Car B" should be kept relatively far apart. This loss makes it easier for subsequent association algorithms (such as Kalman-LSTM) to distinguish and track them. It is a composite loss that combines multiple tasks, taking into account: (A) the core task: the accuracy of detection / localization, which is the main indicator for evaluating the performance of the perception system; (B) cross-modal consistency: ensuring that the output of the single-sensor sub-network and the output of the fused network are as consistent as possible in distribution, achieving stronger robustness and interpretability; (C) metric learning: making the same physical target more compactly embedded under multiple time and multiple sensor observations, and the embedding of different targets more dispersed, which is beneficial for subsequent tracking and trajectory prediction. In other words, the design idea of ​​(10) is to first ensure that the "core detection / localization" task itself performs well (the first item) Then, KL divergence is used for "soft label" alignment, forcing the single-modal subnetwork to converge towards the strong representation after multimodal fusion. Finally, metric constraints are added to further enhance feature discriminativeness, enabling the system to maintain good characteristics of "relative clustering of similar targets and relative separation of different targets" under multi-target, complex lighting, or jitter conditions. This design takes into account "detection accuracy," "multimodal robustness," and "feature discriminativeness," thus maintaining excellent perception and tracking performance in various complex environments (especially low light + jitter, occlusion, rain, fog, etc.). By reasonably adjusting η and ξ, the optimal balance can be found between "classification / localization accuracy," "cross-modal consistency," and "metric learning effect."

[0055] Preferably, in the comprehensive test of three datasets—DarkCityscapes-Night, nuScenes-Night, and the self-built HiLux-100K—the system achieved mAP (0.5:0.95) = 58.6% in nighttime and rain / fog scenarios, which is ≥21% higher than monocular YOLOv8-m; BEV usage IoU = 73.4%; end-to-end inference latency ≤25ms; and average power consumption ≤6W.

[0056] Therefore, the multimodal environment perception system and method for autonomous driving based on the above structure of the present invention have the following beneficial effects:

[0057] (1) This invention integrates the vehicle-mounted imaging subsystem and the 77GHz millimeter-wave radar subsystem, and combines them with the cross-modal Transformer-GAT fusion network to achieve complementary advantages of vision and radar. It solves the problems of quantum efficiency drop and severe motion blur of vision in single-modal perception in scenarios such as night and rain / fog, as well as the limited angular resolution and blurred target shape when radar is used alone, thus improving the reliability of perception in complex environments.

[0058] (2) This invention utilizes the multi-scale Retinex brightness restoration of the AMSR-T module and the IMU-based deblurring branch to simultaneously achieve low-light image enhancement and motion blur elimination. This solves the problem of camera performance in low light (<2 lux) and jitter (>50° / s) conditions. 2 To address the issues of photon noise, motion blur, and exposure drift encountered in complex environments, this paper ensures that the pre-processed image features possess both noise suppression and texture fidelity.

[0059] (3) This invention employs a differentiable homography matrix BEV transform and a self-attention density mask, combined with an improved ST-CFAR algorithm and Kalman-LSTM filtering, to achieve spatiotemporal alignment of cross-modal features and multi-frame target association. This avoids the shortcomings of traditional fusion schemes that rely on lidar or ignore temporal drift, and completes end-to-end inference within a 25ms automotive-grade latency, meeting the real-time requirements of mass production scenarios.

[0060] (4) This invention uses the hardware division of labor between Jetson OrinNX and Zynq UltraScale+MPSoC to allocate deep learning inference and radar signal preprocessing to dedicated computing units, achieving an end-to-end inference latency of less than 25ms and a total system power consumption of no more than 6W.

[0061] (5) This invention introduces radar-visual consistency loss, hybrid training strategy and adversarial domain adapter, combined with high-precision map Bayesian constraints and OTA upgrade mechanism. It solves the spatial consistency problem in cross-modal fusion, enhances the generalization ability of the model in extreme scenarios such as rain, fog and occlusion, and supports online optimization and function iteration, thereby improving the long-term usability and maintainability of the system.

[0062] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the overall system architecture and processing modules;

[0064] Figure 2 This is a schematic diagram of 360° field of view overlap.

[0065] Figure 3 This is a placeholder diagram for a self-attention heatmap;

[0066] Figure 4 This is a schematic diagram of the radar FFT-CFAR-Kalman algorithm.

[0067] Figure 5 A heatmap of cross-attention;

[0068] Figure 6 This is a schematic diagram of the flow of features in Transformer-GAT.

[0069] Figure 7 A diagram illustrating the allocation of hardware resources for GPUs and FPGAs;

[0070] Figure 8 This is a schematic diagram of image enhancement in low-light scenes based on LIME-plus;

[0071] Figure 9 This is a schematic diagram illustrating image enhancement in low-light scenes based on the improved LIME-plus of this invention;

[0072] Figure 10 This is a schematic diagram of the gyroscope angular velocity sequence of the present invention;

[0073] Figure 11 A schematic diagram of the generated adaptive PSF (point spread function);

[0074] Figure 12 For motion-blurred and jittery images (caused by camera or vehicle shaking);

[0075] Figure 13 This is a schematic diagram comparing the low-light processing of the present invention with that of the "Retinex" baseline algorithm;

[0076] Figure 14 This is a schematic diagram of the peak signal-to-noise ratio of the present invention and the "Retinex" baseline algorithm;

[0077] Figure 15 This is a schematic diagram comparing the fuzzy spectrum and the defuzzy spectrum.

[0078] Figure 16 A schematic diagram showing the comparison of edge ringing before and after de-jittering;

[0079] Figure 17 A schematic diagram illustrating the relationship between inference latency and long-side resolution for Enhanced Deformable Video Restoration (Lightweight Version);

[0080] Figure 18 A schematic diagram illustrating the model-predicted control of vehicle trajectory in a congested curve scenario;

[0081] Figure 19 This is a schematic diagram showing the experimental test characteristic curves and average accuracy under different lighting conditions of the present invention.

[0082] Figure 20 The diagram shows the overall performance curves and power consumption. (a) shows mAP vs lu; (b) shows PSNR vs blur; (c) shows latency vs buffer; and (d) shows the power consumption distribution. Detailed Implementation

[0083] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0084] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0085] Example

[0086] like Figure 1 As shown, the present invention provides a multimodal environment perception system for autonomous driving, comprising:

[0087] The system includes an in-vehicle imaging subsystem, an adaptive multi-scale Retinex-Transformer image enhancement module AMSR-T, a 77GHz millimeter-wave radar subsystem, a radar target extraction and tracking module, a bird's-eye view BEV transformation module, a cross-modal Transformer-GAT fusion network, a semantic decision module, and a heterogeneous computing platform.

[0088] Specifically, the hardware includes: a binocular HDR camera (60FPS, 120dB), a 77GHz 4T8RMIMO radar, a BMI088IMU, and a u-bloxZED-F9GNSS-PPS.

[0089] Clock tree: GNSS-10MHz → Programmable PLL → FPGA 250MHz main counter → Frequency divider MIPI-CSI & LVDS trigger.

[0090] Synchronous vibration: 24h high temperature vibration test 3σ=0.83μs.

[0091] The present invention provides a perception method for a multimodal environment perception system for autonomous driving, as detailed below:

[0092] S1, AMSR-T image enhancement

[0093] Multiscale Retinex (MSR) luminance recovery:

[0094] Kernel-PredictionDeblur: IMU integral → kernel radius d (Equation (3)) → SeparableConv.

[0095] Transformer encoder with 6 layers, Embed-Dim64, 8x8 window.

[0096] Color mapping M: Three-layer MLP + Sigmoid.

[0097] Running on: TensorRTFP16, 1280×720 single frame 8.3ms.

[0098] S2, Radar Signal Processing

[0099] Double FFT: 1024×128 points; Hanning window reduces sidelobes by 9.3dB.

[0100] ST-CFAR: Protection / Reference = 24×16 / 32×20; Threshold formula (5).

[0101] Virtual array angle FFT: 64 points; azimuth resolution 1.6°.

[0102] FPGA resources: DSP48E2×640 (<45%), BRAM×312 (<60%).

[0103] Figure 2 The diagram shows the coverage fan-shaped area of ​​a forward-facing binocular HDR camera (horizontal ≈120°) and a 77GHz 4T8RMIMO radar (azimuth ≈150°), with the intersecting area drawn around the vehicle body. Different colored blocks are used to mark the "pure vision zone," "pure radar zone," and "vision-radar overlap zone," highlighting the high-confidence overlap band between the two sensors within 0-60m in front, which is the primary working domain for subsequent cross-modal fusion.

[0104] S3, BEV Alignment and Integration

[0105] Homography matrix: fine-tuned using OpenCV findHomography() with extrinsic parameters; BEV resolution 0.1m.

[0106] Cross-attention: Query = Img, Key / Value = Radar, three layers + Residual.

[0107] GAT: 2 layers, neighborhood radius 0.3m, 38% fewer parameters than pure CNNBEV.

[0108] S4, heterogeneous hardware mapping

[0109] GPU Kernels: K1–K5 total 19.4ms; using TensorCore 128×128 batch GEMM.

[0110] FPGAKernels: FFT_1D / 2D, CFAR, ANG_FFT, DBSCAN; pipeline 5.8ms.

[0111] Memory path:

[0112] Radar→PLDDR4→AXI-DMA→PSDRAM→PCIe→JetsonZero-Copy.

[0113] S5, Training Strategy

[0114] The training of this system can be divided into multiple phases (Phase A / B), and various data augmentation and cross-domain adversarial methods are combined to ensure the robustness and real-time performance of the model under complex conditions. The specific training strategy is as follows:

[0115] S51. Dataset Construction and Preprocessing

[0116] Multimodal data acquisition: Three datasets were used: DarkCityscapes-Night, nuScenes-Night, and the self-built HiLux-100K, covering low-light nighttime conditions, rain and fog conditions, and jitter scenes under different road conditions, respectively. Each frame simultaneously includes a low-light jitter image from the camera and the corresponding 77GHz millimeter-wave radar point cloud (including time-range-Doppler information), and provides grid labels and target instance bounding boxes after BEV projection.

[0117] Label format conversion: The 2D detection boxes in the camera image and the voxel targets extracted by radar are first converted into occupancy raster labels and trajectory annotations in a unified BEV coordinate system. A BEV raster map with a resolution of 0.1m × 0.1m is generated as one of the network inputs.

[0118] Data cleaning and alignment: Based on GNSS-PPS synchronization information and IMU timestamps, multimodal frames are precisely aligned to remove frame-level drift and dropped frames. Samples with excessive jitter are filtered using a threshold (e.g., gyroscope angular velocity integral exceeding 50° / s). 2 (The sample was then labeled as a high-jitter template).

[0119] Preliminary data augmentation: The original low-light jittery image was augmented with conventional methods such as geometric transformation (random cropping, horizontal flipping, micro-rotation ±5°), color perturbation (brightness ±10%, contrast ±15%), and random Gaussian noise (σ = 0.01–0.05); the radar point cloud was augmented with random rotation (±5°), random occlusion (simulating 5% of local failure points), and random noise (with a deviation of ±0.1m in each dimension).

[0120] S52, Phased Training (Phase A / B)

[0121] S521, Phase A: Single-modal pre-training

[0122] Visual branch pre-training: Using only low-light jittered images from DarkCityscapes-Night, the images are enhanced by the AMSR-T module (multi-scale Retinex+Transformer+IMU-driven deblurring) defined in Equations (1)–(3). Figure 3 This section presents a heatmap of attention weights overlaid on the final layer of the Transformer encoder, using an enhanced low-light image as the base map. Red highlighted areas correspond to key targets such as pedestrians / vehicle lights, while blue areas represent suppressed background. The sidebar shows the 8×8 window partitioning and the stacked layers of the 6-layer encoder, demonstrating the significant attention regions of AMSR-T in low-light jitter scenes. The visual backbone employs a pre-trained lightweight backbone network (such as EfficientVit-M0 or FasterNet), fine-tuned on the visible light annotation set DarkCityscapes-Night, enabling the visual branches to recover usable deep feature representations.

[0123] Radar branch pre-training: In the nuScenes-Night and HiLux-100K datasets, using only radar voxels and their corresponding target detection labels, target points are extracted using the ST-CFAR-Kalman module. Based on PointPillars or an improved lightweight radar network (such as SECOND-Lite), target detection and trajectory pre-prediction are performed on the radar point cloud to obtain preliminary radar feature branches. Figure 4 The process flow is shown from left to right: Original ADC → RangeFFT (1024) → DopplerFFT (128) → Time-Range-Doppler Cube → Improved ST-CFAR (Protection / Reference = 24×16 / 32×20, Threshold Formula (4)) → Centroid-Voxel Clustering → Kalman-LSTM Prediction / Update. Each step is labeled with FPGA resource usage and output dimension; key indicators such as azimuth resolution of 1.6° and pipeline delay of 5.8ms are given at the bottom.

[0124] Single-modal, single-task loss: The loss function for the vision branch is the conventional detection loss (λ_cls·focal + λ_loc·IoU); the loss for the radar branch is the point cloud target detection loss (including classification and regression parts). At this stage, cross-modal consistency or triplet loss is not introduced; the focus is solely on achieving high accuracy for each branch under single-modal conditions (visual mAP≈42%, radar branch mAP≈38%).

[0125] S522, Phase B: Multimodal Fusion Fine-tuning

[0126] Alignment and fusion network initialization: Load the pre-trained vision and radar branch parameters in Phase A to construct a cross-modal Transformer-GAT network.

[0127] First, coarse alignment and fine-tuning were performed on the joint dataset of DarkCityscapes-Night and nuScenes-Night. The input to the fusion network was: BEV image raster nodes (visual branch extraction) and radar voxel nodes (radar branch projected onto BEV). Figure 5 Using the BEV grid as the coordinate system, pseudo-color is used to display the attention weights of the cross-modal Transformer-GAT in the "image grid → radar voxel" direction: warm colors represent high weights (visual-radar high consistency units), and cool colors represent low weights. The node cosine similarity formula (6) and temperature coefficient T are given on the side of the figure to illustrate the attention normalization process.

[0128] Hybrid loss design: The hybrid loss defined in equation (10) is adopted, and its expression is as follows.

[0129]

[0130] The first two terms are the basic detection loss (aggregated training of the visual and radar branches); the third term is the cross-modal consistency constraint (KL divergence), which sets the single-modal prediction distribution p sensory Towards a more reliable p after fusion fusion The fourth term is the triplet loss, used to aggregate the joint feature representation of the same target across multiple visual and radar frames and to push forward the representations of different targets. The initial hyperparameters are set to η = 0.1, ξ = 0.1, and α = 0.3.

[0131] Adversarial Domain Adaptation (CycleGAN): For rain and fog occlusion scenarios (simulating the DarkCityscapes-Night challenge), the CycleGAN adversarial domain adaptation module is introduced. It maps the style domain of the DarkCityscapes visual image to the rain and fog domain in an unsupervised manner, generating synthetic training samples. Multimodal fusion training is then performed on these samples to enhance the model's robustness to visual features in foggy and rainy scenes.

[0132] Quantization-Aware Training (PTQ): Once the fusion network achieves approximately 56% multimodal mAP, it enters the later stage of quantization-aware training. During this stage, the Transformer-GAT part is quantized in 8-bit hierarchical manner, while the visual and radar backbones retain FP16 operations. Dynamic range calibration (QAT) is used to fine-tune the output to ensure that the overall accuracy decrease after quantization is less than 1%. Figure 6 The diagram illustrates a dual-branch parallel structure: the left side shows "Image Patch Embedding → Encoder Module × L → Graph Attention Layer × K → Multilayer Perceptron Head," while the right side shows the radar voxel isomorphism process. The two branches are cross-connected before and after the GAT layer via cosine attention. The feature dimension (Embed-Dim64) and window size (8×8) are labeled next to the arrows, and the residual connections and Layer-Norm locations are listed at the bottom.

[0133] Learning rate and iteration strategy: Segmented learning rate scheduling (Warm-up + Cosine Annealing) is adopted: the first two epochs linearly increase the learning rate from 1e... -5 Warmup up to 1e-4; Cosine Annealing adopted in a subsequent total of 50 epochs from 1e -4 decay to 1e -6 During the quantization perception phase, fine-tune the learning rate for another 10 epochs, starting from 5e. -5 Gradually decay to 1e -6 Each batch is 16 in size (consisting of a pair of visual frames + a radar frame), using the SGD optimizer with a momentum of 0.9 and a weight decay of 1e. -4 .

[0134] S523, Training Process and Indicator Generation

[0135] During training, the script fig_scripts / create_FigH_metrics.m automatically reads metrics_table.mat and outputs ROC curves, mAP histograms, and system power consumption line graphs in real time, saving them to the results / fig_H / directory to help researchers observe the performance curve changes of the model under different lighting, rain, fog, and shaking intensities.

[0136] S524. Early Termination and Model Selection

[0137] In Phase B, early stopping is triggered when the validation set mAP (IoU 0.5: 0.95) fluctuates by less than 0.2% over 5 consecutive epochs without a decrease in power consumption, and the current optimal weights are saved as the final model. The model has approximately 23.8M parameters and inference computation (FLOPs) of approximately 68.3G.

[0138] S6, Performance Evaluation

[0139] To comprehensively evaluate the detection, tracking, and real-time performance of this invention in different scenarios, multi-dimensional performance tests were designed, and the results are presented in detail in Figures 8-20. The specific evaluation content is as follows:

[0140] S61. Datasets and Evaluation Metrics

[0141] Datasets: Tests were conducted on three datasets: Dark City scapes-Night, nuScenes-Night, and HiLux-100K, to ensure coverage of low-light conditions at night, variable rain and fog conditions, and various road surface vibrations.

[0142] S62. Evaluation Indicators:

[0143] Detection accuracy (mAP@0.5:0.95): Target categories include vehicles, pedestrians, bicycles, etc., with IoU thresholds ranging from 0.5 to 0.95, in increments of 0.05.

[0144] BEV Occupancy Map IoU: For the raster map projected from the BEV, the target segmentation and localization accuracy is measured in pixel-level IoU (Intersection over Union).

[0145] End-to-end inference latency: Measured on the Jetson OrinNX+Zynq UltraScale+ZU7EV heterogeneous platform, including: total latency of image enhancement (AMSR-T), radar processing (ST-CFAR-Kalman), cross-modal fusion (Transformer-GAT), and final detection output, in milliseconds.

[0146] System power consumption: Average power consumption during full system (Jetson GPU+FPGA) operation, measured using a power analyzer (e.g., TIUSB-C-Power Scope), in watts. Figure 7The upper half of the display shows a Jetson OrinNX board, responsible for core deep learning operators such as convolutional feature extraction, multi-head attention, and batch normalization; the lower half shows a Zynq UltraScale+ZU7EV FPGA, handling ST-CFAR, Kalman updates, and 8-bit quantization matrix multiplication. The two boards are interconnected via PCIe+AXI-DMA zero-copy. The right-hand bar shows the DSP48 (<45%) and BRAM (<60%) utilization rates, and also indicates that the total latency of the five Jetson CUDA kernels is 19.4ms.

[0147] Tracking consistency: IDF1 (ID accuracy) and MOTA (Multiple Object Tracking Accuracy) were used to evaluate the stability of continuous frame tracking and ID retention in crowded scenes (such as Scene 18).

[0148] S63, Image Enhancement Effects Figure 8 & Figure 9 )

[0149] Figure 8 The enhancement effect of LIME-plus in low-light scenes is demonstrated: as can be seen from the image comparison, LIME-plus can significantly improve contrast under low light conditions, but it is insufficient in terms of texture fidelity and noise suppression.

[0150] Figure 9 This paper showcases the enhanced results of this invention based on an improved LIME-plus (combining multi-scale Retinex and dynamic adjustment of Transformer attention weights): Compared to LIME-plus, the improved method not only improves contrast in low-light scenes (average PSNR improvement of 2.3dB and SSIM improvement of 0.076), but also better preserves detail texture and suppresses noise. This demonstrates the effectiveness of AMSR-T in the preprocessing stage, making subsequent feature detection more robust.

[0151] S64, Radar de-jitter and de-blurring performance ( Figure 10 – Figure 16 )

[0152] Figure 10 The angular velocity sequence curves of the IMU gyroscope collected by this invention show that the angular velocity of the vehicle changes drastically under shaking conditions (the peak value can reach ±60° / s).

[0153] Figure 11 The diagram shows the adaptive PSF (point spread function) generated based on formula (3). As the integral of angular velocity increases, the PSF radius changes dynamically, generating deblurring convolution kernels of different shapes for jitter of different intensities.

[0154] Figure 12 The comparison between the original jittered image and the image after jitter removal shows that the present invention effectively restores image edge details under extreme jitter conditions, and significantly reduces edge ringing.

[0155] Figure 13 and Figure 14 Comparing the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) curves of the present invention and the traditional Retinex in low-light processing, the present invention improves the average PSNR by 4.7 dB and the SSIM by 0.084.

[0156] Figure 15 and Figure 16 The comparison between the blurred spectrum and the clear spectrum before and after frequency domain deblurring is shown, along with the corresponding edge ringing suppression effect, verifying the superiority of combining Wiener complex convolution and EdgeTaper from a frequency domain perspective.

[0157] S65, Target Detection and Tracking Performance Figure 17 – Figure 19 )

[0158] Figure 17 The inference latency curves of the enhanced deformable video recovery (lightweight version) are shown at different long-side resolutions: as the input resolution gradually increases from 640×360 to 1920×1080, the AMSR-T inference latency increases from 6.3ms to 12.8ms, remaining controllable (above 16fps), ensuring that the system can preprocess image features within the end-to-end real-time time window.

[0159] Figure 18 This diagram illustrates the MPC (Model Predictive Control) vehicle trajectory in a congested curve scenario. The experiment records the comparison between the detection results of this invention and the MPC planned trajectory, verifying that the model can output accurate target positions in real time for subsequent longitudinal and lateral control under complex curves.

[0160] Figure 19 The presentation shows the variation curves of the model detection accuracy (Average Precision, AP) under different lighting conditions, along with the corresponding power consumption distribution. It can be seen that the AP reaches 87.2% in open daytime scenes (light intensity > 20 lux); and the fusion branch accuracy is 58.6% in low-light nighttime conditions (< 2 lux), representing an improvement of ≥ 21% compared to monocular YOLOv8-m; power consumption is controlled at 5.8W ± 0.3W, fully meeting the requirements for automotive-grade edge deployment.

[0161] S66. Comprehensive Performance and Comparative Analysis Figure 20 )

[0162] Figure 20 For the overall performance curves and system power consumption distribution: Figure 20(a) in the figure shows mAP vs lux; Figure 20 (b) shows PSNR vs blur; Figure 20 (c) in the diagram illustrates delay vs. caching; Figure 20 (d) in the diagram shows the power consumption distribution.

[0163] As can be seen, the horizontal axis represents different detection scenarios (daytime / nighttime / rain / fog / shaking), the left side of the vertical axis represents mAP@0.5:0.95, and the right side of the vertical axis represents power consumption (W). In all test scenarios, this invention achieves mAP≥55% and power consumption is consistently below 6W.

[0164] Compared with current academic SoTA (such as BEVFusion+FusionTransformer), the detection accuracy is comparable, while the power consumption is reduced by approximately 48%.

[0165] Comparative Experiments: Our method was further tested on the same hardware platform against two mainstream multimodal fusion schemes (FusionNetA and FusionNetB). In the mAP comparison under low light and jitter scenarios, our method achieved 58.6% vs. FusionNet A 43.2% vs. FusionNet B 46.5%, significantly outperforming FusionNet A. Average inference latency was 23ms vs. FusionNet A 38ms vs. FusionNet B 41ms. Average power consumption was 5.9W vs. FusionNet A 9.8W vs. FusionNet B 10.2W.

[0166] Specifically, the consistency assessment includes:

[0167] Three high-speed jitter scene videos were selected in the HiLux-100K, and the IDF1 and MOTA metrics were evaluated respectively. The average IDF1 of this invention was 75.3% and MOTA was 68.9% across the three videos; while the baseline of a single camera in the unfused scene was only IDF1 = 52.1% and MOTA = 47.8%, indicating that cross-modal fusion significantly improves the consistency and tracking stability of targets between consecutive frames.

[0168] Post-deployment stability testing, specifically:

[0169] After 12 hours of continuous operation using a heterogeneous combination of Jetson OrinNX and Zynq UltraScale+ZU7EV boards, the system's measured performance (mAP) decreased from 58.6% in the first hour to 57.9% in the 12th hour, while power consumption increased from 5.8W to 6.1W (an average increase of 0.4W). Good stability and anti-drift capabilities were maintained.

[0170] Through the above multi-dimensional evaluation, it can be seen that the present invention can still maintain high-precision detection (mAP≥58%), high tracking consistency (IDF1≥75%), ultra-low latency (≤25ms) and extremely low power consumption (≤6W) even in extreme scenarios of "night, rain, fog and jitter", demonstrating significant advantages in academic and industrial applications.

[0171] In summary, the multimodal environment perception system and method provided by this invention have the following technical effects:

[0172] Robustness: mAP 58.6% in nighttime rain and fog, an improvement of 21%–82% over YOLOv8-m.

[0173] Real-time performance: Jetson+PL total latency 25ms; power consumption <6W, meeting automotive-grade AEC-Q100 standards.

[0174] Cost: Compared to BEVFusion (32L LiDAR), the BOM cost is reduced by approximately 75%.

[0175] Therefore, this invention deeply couples low-light jitter adaptive enhancement with radar-vision BEV fusion, and completes end-to-end inference within 25ms through GPU-FPGA collaborative acceleration, with system power consumption <6W. Its performance surpasses existing academic SoTA and industrial solutions, and all results can be directly reproduced in an Ubuntu 22.04 + CUDA 12.3 + Vivado 2023.2 environment. It possesses high added value and economic practical value.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal environment perception system for autonomous driving, characterized in that: include: The vehicle-mounted imaging subsystem is used to acquire raw image sequences; The adaptive multi-scale Retinex-Transformer image enhancement module is an AMSR-T module, used to perform low-light enhancement and jitter reduction processing on the original image sequence and output an enhanced image; The AMSR-T module includes a deblurring branch based on the angular velocity estimation of the gyroscope inertial measurement unit (IMU), and the predicted fuzzy kernel formula is as follows: ； In the formula, 𝑘 𝑡 The predicted blur kernel is dynamically adjusted based on the gyroscope's angular velocity to remove motion blur from the image; where 𝑢, 𝑣 are the kernel coordinates, 𝜃 𝑡 The cumulative rotation angle within the frame exposure window; the above formula expresses the variable dependency of the PSF point spread function generated by the inertial measurement unit (IMU); the angular velocity output by the IMU sensor. It is used to estimate the motion of a vehicle or camera, and thus calculate the size of the blur kernel; 𝑡 The kernel reflects the degree of motion blur in the image and is used in subsequent deblurring processing; Its convolution kernel size satisfies: d = ； in, This is an empirical proportionality coefficient. Let V be the IMU's three-axis angular velocity vector at time φ. The 77GHz millimeter-wave radar subsystem is used to output a time-range-Doppler three-dimensional cube for inter-frame registration; The radar target extraction and tracking module is used to process the three-dimensional cube, extract and track targets, and obtain a target list; The radar target extraction and tracking module employs an improved spatiotemporal constant false alarm rate (CFAR) algorithm, ST-CFAR, with a threshold of [missing information]. 𝑇(𝑟,𝑓 d )=𝜇(𝑟,𝑓 d )+𝛼𝜎(𝑟,𝑓 d ); It also combines Kalman-LSTM filtering to complete multi-frame target association and trajectory smoothing; 𝑇(𝑟,𝑓 d The threshold of the radar signal is used for target extraction and filtering; 𝜇(𝑟,𝑓 d The mean of the radar signal; φ constant, representing the gain coefficient of the threshold; φ(φ, φ) d The standard deviation of a radar signal represents its fluctuation or noise amplitude; the above formula defines the adaptive threshold 𝑇(𝑟,𝑓) for the radar signal. d Its calculation is based on the mean and standard deviation of radar signals; by setting an appropriate gain coefficient φ, the effective distinction between targets and noise can be achieved. A bird's-eye view BEV transformation module is used to project the enhanced image and the target list onto the same BEV reference frame; A cross-modal Transformer-GAT fusion network is used to perform graph attention fusion on BEV image raster nodes and radar voxel nodes, and output joint features. The semantic decision module is used to output target category, location, and trajectory information based on the joint features; A heterogeneous computing platform is used to achieve an end-to-end inference latency of less than 25ms and a total system power consumption of no more than 6W.

2. The multimodal environment perception system for autonomous driving according to claim 1, characterized in that: The BEV transformation module projects image features onto BEV grid coordinates using a differentiable homography matrix and fills projection holes using a self-attention density mask.

3. The multimodal environment perception system for autonomous driving according to claim 1, characterized in that: The cross-modal Transformer-GAT fusion network calculates node attention weights using cosine similarity, thereby achieving cross-modal alignment and fusion of image and radar features.

4. A multimodal environment perception system for autonomous driving according to claim 3, characterized in that: The cross-modal Transformer-GAT fusion network calculates node attention weights using cosine similarity, thereby achieving cross-modal alignment and fusion of image and radar features.

5. A multimodal environment perception system for autonomous driving according to claim 4, characterized in that: The target detection loss function of the system includes classification loss, location loss, and radar-vision consistency loss, which optimizes cross-modal detection accuracy and spatial consistency.

6. A perception method for a multimodal environment perception system for autonomous driving according to any one of claims 1-5, characterized in that: Includes the following steps: S1. Acquire the original image sequence and the time-range-Doppler 3D cube of the 77GHz millimeter-wave radar; S2. The original image sequence is subjected to low-light enhancement and jitter removal processing through an adaptive multi-scale Retinex-Transformer network to obtain the enhanced image; S3. Process the three-dimensional cube to extract and track targets, and obtain a target list; S4. Project the enhanced image and the target list onto the same BEV reference frame; S5. Perform graph attention fusion on the grid nodes and radar voxel nodes of the BEV image through the cross-modal Transformer-GAT fusion network to generate joint features; S6. Output the target category, location, and trajectory information based on the joint features.

7. The perception method for a multimodal environment perception system for autonomous driving according to claim 6, characterized in that: Step S2 involves low-light enhancement and jitter reduction of the original image sequence, including multi-scale Retinex brightness restoration and motion blur kernel prediction based on the gyroscope inertial measurement unit.

8. The perception method for a multimodal environment perception system for autonomous driving according to claim 6, characterized in that: The step of processing the 3D cube in step S3 includes using the ST-CFAR algorithm for target extraction and using Kalman-LSTM filtering to achieve multi-frame target tracking.

Images