Vehicle-mounted AI visual identification terminal in intelligent driving scene

Abstract

The invention relates to the technical field of intelligent driving, in particular to a vehicle-mounted AI visual recognition terminal in an intelligent driving scene, which comprises a space-time alignment module, a physical illumination recovery module, a self-adaptive recognition module and a decision fusion module. The space-time alignment module realizes sampling time difference compensation and space remapping of camera images and millimeter wave radar point cloud data; the physical illumination recovery module adopts a physical parameterized illumination model fusing an atmospheric scattering equation and a camera response function, and reconstructs real details of an overexposure / underexposure region in combination with an illumination transmission equation and a Monte Carlo ray tracing algorithm; the adaptive recognition module calculates topology complexity according to the curvature change rate of the lane line and the spatial distribution density of the traffic signs, and dynamic loading and efficient recognition of a recognition model are achieved; and the decision fusion module rejects a false detection target in combination with space-time consistency verification. The method has high recognition precision and high decision reliability in complex illumination and high-dynamic traffic environments, and is suitable for an intelligent driving system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, and in particular to a vehicle-mounted AI visual recognition terminal in an intelligent driving scenario. BACKGROUND

[0002] In the field of intelligent driving, the vehicle-mounted perception system, as the core component of automatic driving decision control, undertakes the real-time perception task of the surrounding environment of the vehicle. The current mainstream scheme generally adopts a multi-sensor fusion structure to cooperatively complete the recognition and tracking of key targets such as traffic signs, lane lines, vehicles and pedestrians by using heterogeneous perception devices such as cameras, laser radars and millimeter wave radars. Among them, visual perception is widely used in lane line detection and semantic segmentation tasks due to its rich image information and high resolution; laser radar has high-precision ranging capability and can be used for target three-dimensional positioning and trajectory prediction. However, in the actual driving process, due to the inconsistency of sensor sampling frequency, the sharp change of light, the large difference in complexity of traffic scene topology, and the existence of isolated false detection of recognition results, the overall recognition stability and decision robustness of the system are often reduced, which is difficult to meet the high-precision and low-error vehicle-mounted perception requirements in complex dynamic environments.

[0003] The existing technology generally has three outstanding problems when processing multi-source perception data fusion: first, there is a sampling time difference between the camera image and the millimeter wave or laser radar, which fails to achieve high-precision spatio-temporal alignment, resulting in asynchronous dynamic targets in the visual image and point cloud trajectories, affecting the subsequent fusion performance; second, under extreme light conditions such as strong light, shadow, rain and fog, the visual image is prone to overexposure or underexposure areas, and existing image enhancement-based algorithms cannot restore its physical reality, thereby reducing the target detection accuracy; third, the current recognition model is statically configured and does not dynamically adjust the computing resources according to different scene complexities, and lacks a false detection target elimination mechanism based on cross-modal trajectory association, resulting in non-physical consistent errors in the recognition output, affecting the final control decision of the autonomous driving system. SUMMARY

[0004] The present application provides a vehicle-mounted AI visual recognition terminal in an intelligent driving scenario, which has high spatio-temporal synchronization capability, strong light recovery capability, recognition adaptive capability and cross-modal error screening mechanism to meet the reliable perception requirements in an intelligent driving scenario.

[0005] A vehicle-mounted AI visual recognition terminal in an intelligent driving scenario, comprising a spatio-temporal alignment module, a physical light recovery module, an adaptive recognition module and a decision fusion module, wherein;

[0006] The space-time alignment module is configured to acquire camera raw images and millimeter wave radar point cloud data, generate a moving target optical flow field based on the radar point cloud data, compensate for a sampling time difference between the camera and the radar through flow field back projection, and output a space-time aligned image.

[0007] The physical light recovery module inputs the space-time aligned image into a physical parametric light model, the physical parametric light model fuses an atmospheric scattering equation and a camera response function, reconstructs physical real details of overexposed / underexposed areas by solving a light transmission equation, and outputs a light recovery image.

[0008] The adaptive recognition module performs road structure topology analysis on the light recovery image, extracts lane line curvature change rate and traffic sign spatial distribution density, dynamically loads a lightweight recognition model or a high-precision recognition model according to topology complexity, and outputs a scene topology adaptive recognition result.

[0009] The decision fusion module cross-modal correlates the scene topology adaptive recognition result with laser radar target trajectory prediction, filters out isolated false detection targets through space-time consistency verification, and outputs a final recognition result.

[0010] Optionally, the space-time alignment module includes:

[0011] The camera raw images and the millimeter wave radar point cloud data are collected simultaneously.

[0012] The millimeter wave radar point cloud data is clustered and segmented to extract Doppler velocity vectors of each moving target, and the Doppler velocity vectors of all moving targets are superimposed to generate a dense moving target optical flow field.

[0013] According to a difference between a camera exposure timestamp and a millimeter wave radar scanning period, a dynamic interpolation weight of the moving target optical flow field is calculated.

[0014] The moving target optical flow field is back projected to the camera raw images according to the dynamic interpolation weight, and the camera raw images are remapped through bilinear sampling.

[0015] The remapped camera raw images are edge-smoothed to generate a space-time aligned image and output to the physical light recovery module.

[0016] Optionally, the physical light recovery module includes:

[0017] The space-time aligned image output by the space-time alignment module is received.

[0018] The space-time aligned image is input into a physical parametric light model, wherein the physical parametric light model is constructed by fusing an atmospheric scattering equation and a camera response function.

[0019] The atmospheric scattering coefficient of the scene is calculated using the atmospheric scattering equation, and the atmospheric scattering coefficient is adaptively optimized based on the sky region segmentation results.

[0020] Nonlinear response correction is performed on the spatiotemporally aligned image using the camera response function to generate a sensor characteristic calibration image.

[0021] Optionally, the physical illumination restoration module further includes:

[0022] By substituting the atmospheric scattering coefficient and sensor characteristics into the irradiance transfer equation, the Monte Carlo ray tracing algorithm is used to solve the radiative transfer in overexposed and underexposed areas, and the physical real details are reconstructed.

[0023] The reconstructed image is subjected to gradient domain fusion processing to generate an illumination restoration image, which is then output to the adaptive recognition module.

[0024] Optionally, the adaptive recognition module includes:

[0025] Receive the illumination restoration image output by the physical illumination restoration module;

[0026] Perform road structure topology analysis on the illumination restoration image to extract the lane centerline equation;

[0027] Calculate the rate of change of lane curvature per meter of path length based on the lane centerline equation;

[0028] Detect the bounding boxes of traffic signs in the illuminated image and count the area per unit area (10m²). 2 The spatial distribution density of traffic signs within the area;

[0029] Calculate the topological complexity based on the rate of change of lane curvature and the spatial distribution density of traffic signs:

[0030] Topological complexity = 0.7 × lane line curvature change rate + 0.3 × traffic sign spatial distribution density.

[0031] Optionally, the step of dynamically loading a lightweight recognition model or a high-precision recognition model based on topological complexity and outputting scene topology adaptive recognition results includes:

[0032] If the topology complexity is less than the threshold, a lightweight recognition model is loaded; if the topology complexity is greater than or equal to the threshold, a high-precision recognition model is loaded.

[0033] The illumination-restored image is input into the loaded recognition model for target detection and semantic segmentation, generating scene topology adaptive recognition results and outputting them to the decision fusion module.

[0034] Optionally, the decision fusion module includes:

[0035] Receive the scene topology adaptive recognition result output by the adaptive recognition module, and simultaneously acquire the target trajectory prediction data of the lidar;

[0036] The scene topology adaptive recognition results are cross-modal correlated with the LiDAR target trajectory prediction data, and the visual target box and LiDAR trajectory points are matched by the Hungarian algorithm to generate correlation matching pairs.

[0037] Spatiotemporal consistency verification is performed on the associated matching pairs, and the position offset and motion direction deviation of the visual target and the lidar trajectory are calculated within 5 consecutive frames.

[0038] Optionally, the decision fusion module further includes:

[0039] If the position offset is greater than 1.5 meters or the movement direction deviation is greater than 15 degrees, the target is determined to be an isolated false detection target and a set of non-physically feasible targets is generated.

[0040] The system filters out targets not included in the physically feasible target set from the scene topology adaptive recognition results, generates the final recognition result, and outputs it to the vehicle control system.

[0041] Optionally, acquiring the lidar target trajectory prediction data includes:

[0042] Collect raw point cloud data from lidar and perform point cloud denoising through voxel grid downsampling;

[0043] Euclidean clustering was performed on the denoised raw point cloud data of the lidar to separate independent moving target point cloud clusters.

[0044] Perform multi-target tracking on each moving target point cloud cluster for 5 consecutive frames to generate multi-target tracking trajectory;

[0045] Based on the multi-target tracking trajectory, the motion state of each target is estimated by a Kalman filter predictor, and the motion state includes position coordinates, velocity vector and acceleration vector;

[0046] Establish a kinematic model of the target based on its motion state, and predict the target trajectory point sequence within the next 100ms.

[0047] By integrating the trajectory point sequences of all targets, target trajectory prediction data for lidar is generated.

[0048] The beneficial effects of this invention are:

[0049] This invention utilizes a spatiotemporal alignment module to dynamically compensate for the time difference and spatially remap camera images and millimeter-wave radar point cloud data. Combined with a back-projection correction mechanism based on optical flow fields, it effectively solves the visual error problem caused by asynchronous sampling from multiple sensors. During image resampling, this module employs bilinear interpolation and edge smoothing, significantly enhancing image clarity and structural stability in dynamic scenes, providing highly consistent input for subsequent recognition tasks, and is particularly suitable for high-speed driving and heavily occluded environments.

[0050] This invention constructs a physically parameterized illumination model that integrates atmospheric scattering equations and camera response functions through a physical illumination restoration module. Based on a Monte Carlo ray tracing algorithm, it reconstructs the radiative transfer process in overexposed and underexposed areas, effectively restoring true details lost in images due to factors such as strong light, shadows, rain, and fog. Combined with gradient domain fusion processing, this module can output images with high contrast, uniform brightness, and structural consistency, significantly improving the recognition accuracy and adaptability of the visual perception module in complex weather and lighting scenarios.

[0051] This invention achieves dynamic switching of the recognition model based on scene topology complexity through an adaptive recognition module, effectively balancing computational resource consumption and recognition accuracy, and adapting to different traffic densities and road morphologies. Combined with the cross-modal target trajectory association and spatiotemporal consistency verification mechanism introduced by the decision fusion module, it utilizes the Hungarian algorithm to match visual bounding boxes and radar trajectories, and incorporates continuous motion consistency indices to eliminate isolated false detections, thereby significantly reducing the visual false recognition rate and outputting a structurally complete and highly reliable final recognition result. This effectively improves the safety and stability of the vehicle control system in dynamic traffic environments. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Fig. 1 This is a schematic diagram of the terminal process according to an embodiment of the present invention;

[0054] Fig. 2 This is a schematic diagram of the decision fusion module in an embodiment of the present invention. Detailed Implementation

[0055] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should also be noted that, to make the embodiments more comprehensive, the following embodiments are the best and preferred embodiments, and those skilled in the art can use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0056] It should be noted that the use of terms such as "an embodiment," "an embodiment," "an exemplary embodiment," and "some embodiments" in the specification indicates that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0057] Generally, terms can be understood at least partly from their use in context. For example, depending at least partly on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in a singular sense, or a combination of features, structures, or characteristics in a plural sense. Additionally, the term "based on" can be understood not necessarily to convey an exclusive set of factors, but rather, alternatively, depending at least partly on the context, to allow for the presence of other factors that are not necessarily explicitly described.

[0058] like Figs. 1-2 As shown, an in-vehicle AI visual recognition terminal for intelligent driving scenarios includes a spatiotemporal alignment module, a physical illumination recovery module, an adaptive recognition module, and a decision fusion module, wherein;

[0059] The spatiotemporal alignment module is used to acquire the original images from the camera and the point cloud data from the millimeter-wave radar. Based on the radar point cloud data, it generates the optical flow field of the moving target, compensates for the sampling time difference between the camera and the radar through back projection of the flow field, and outputs the spatiotemporal aligned image.

[0060] The physical illumination restoration module inputs the spatiotemporally aligned image into the physically parameterized illumination model. The physically parameterized illumination model integrates the atmospheric scattering equation and the camera response function, and reconstructs the physical real details of the overexposed / underexposed areas by solving the illumination transport equation, and outputs the illumination restoration image.

[0061] The adaptive recognition module performs road structure topology analysis on the illumination restoration image, extracts the lane line curvature change rate and traffic sign spatial distribution density, dynamically loads a lightweight recognition model or a high-precision recognition model according to the topological complexity, and outputs scene topology adaptive recognition results.

[0062] The decision fusion module performs cross-modal correlation between the scene topology adaptive recognition results and the LiDAR target trajectory prediction, filters out isolated false detection targets through spatiotemporal consistency verification, and outputs the final recognition results.

[0063] The spatiotemporal alignment module includes:

[0064] 1. Simultaneous acquisition of raw camera images and millimeter-wave radar point cloud data: The visual recognition terminal is equipped with an integrated synchronization device to simultaneously trigger the image acquisition device and the millimeter-wave radar module. The images acquired by the camera are two-dimensional image frames in RGB format, labeled with an exposure timestamp T. cam Millimeter-wave radar point cloud data contains three-dimensional spatial coordinate information (x, y, z). i ,y i ,z i ) and the corresponding Doppler velocity vector Includes a scan cycle timestamp T radar .

[0065] 2. Moving target clustering and segmentation of millimeter-wave radar point cloud data: First, density clustering algorithms (such as DBSCAN) are used to cluster the point cloud data to form a set of moving targets. For each moving target O j Extract its corresponding Doppler velocity vector Its center velocity can be calculated as follows:

[0066]

[0067] 3. Generate a dense optical flow field for moving targets: Map the Doppler velocity vectors of all targets onto the camera image plane according to their two-dimensional projections to form a dense optical flow vector map. The optical flow vector of each pixel (x, y) is derived from the nearest moving target. Values ​​can be assigned or generated through bilateral interpolation.

[0068] 4. Calculate the dynamic interpolation weights for the optical flow field of a moving target: To compensate for the sampling time difference between the camera image and the millimeter-wave radar, calculate the time difference ΔT = T. cam -T radar And based on this, the optical flow interpolation weighting factor α is calculated as follows:

[0069]

[0070] Where T radar-cycle This represents the complete scan cycle of a millimeter-wave radar.

[0071] 5. Backproject the optical flow field of the moving target onto the original image from the camera: Apply the following backprojection model to each pixel (x,y) in the image according to the interpolation weight α:

[0072]

[0073] in Let x represent the components of the optical flow field along the x and y axes.

[0074] Bilinear sampling is used to remap the original image to obtain the corresponding transformed image I′(x,y).

[0075] 6. Edge Smoothing and Image Correction: Gaussian filters are used to smooth the edges of abrupt changes in the remapped image, avoiding jagged edges or blurring caused by pixel movement, and generating the final spatiotemporally aligned image I. aligned (x,y).

[0076] 7. Output results to the Physical Illumination Restoration module: Align the spatiotemporally image I... aligned (x,y) is passed as standard input to the physical illumination recovery module for subsequent reconstruction of physical details in overexposed / underexposed areas.

[0077] Example explanation:

[0078] Assume the current camera capture timestamp is T. cam =100.05ms, the previous scan period of the millimeter-wave radar is T radar =100.00ms, scan period is T radar-cycle =20.00ms, then:

[0079]

[0080] This indicates that the radar data lags very little relative to the image in this scenario, and therefore can be corrected by slight interpolation.

[0081] The physical lighting restoration module includes:

[0082] 1. Receive spatiotemporally aligned image: First, receive image I output from the spatiotemporally aligned module. aligned (x,y) This image has been aligned and corrected with the millimeter-wave radar point cloud in both temporal and geometric space, and has a high degree of motion consistency.

[0083] 2. Input processing of physically parameterized lighting model: I aligned The (x,y) input is a physically parameterized illumination model constructed by fusing the atmospheric scattering equation and the camera response function. This model jointly considers the attenuation of illumination by the atmospheric environment and the nonlinear characteristics of device imaging, and has the ability to model realistically.

[0084] 3. Calculate the atmospheric scattering coefficient of the scene: Estimate the atmospheric scattering intensity in the scene using an atmospheric scattering model, based on the following atmospheric scattering equation:

[0085] I(x,y)=J(x,y)·t(x,y)+A·(1-t(x,y));

[0086] Where I(x,y) is the observed image brightness, J(x,y) is the actual scene brightness (to be recovered), A is the global atmospheric light, and t(x,y) = e -βd(x,y) denoted as the transmission rate, β as the atmospheric scattering coefficient, and d(x,y) as the depth estimate.

[0087] For the sky region, use an image segmentation network such as SkyNet to extract its mask region. And perform adaptive optimization of scattering coefficients within it:

[0088]

[0089] 4. Camera response function nonlinearity correction: For image I aligned (x,y) Apply the pre-calibrated camera response function f -1 (·), Generate sensor characteristic calibration image:

[0090] I calib (x,y)=f -1 (I aligned (x,y));

[0091] Where f is the pixel response curve of the device under exposure control (usually modeled using a polynomial or S-curve approximation).

[0092] 5. Substitute into the light transmission equation and solve for the radiative transfer: Compare the scattering coefficient β with the sensor characteristic calibration image I. calib Substitute them into the light transmission equation:

[0093] L out (x,y)=∫ Ω L in (x,y,ω)·e -βd(x,y) dω;

[0094] Among them, L in Let ω be the incident radiation, Ω be the direction of the ray, and L be the omnidirectional unit sphere. out Reconstruct brightness for the output image.

[0095] The integral above is approximated by sampling using the Monte Carlo ray tracing algorithm (typically using N=128 paths for random directional sampling), and expressed as:

[0096]

[0097] 6. Reconstruct the image and perform gradient domain fusion: The resulting brightness image exhibits local transition jumps. To eliminate block artifacts and boundary artifacts, gradient domain fusion is applied.

[0098]

[0099] The objective function can be solved using the Poisson equation to ensure the continuity of the image structure and its physical consistency.

[0100] 7. Output to the adaptive recognition module: the final illumination restoration image I fused (x,y) possesses physical compensation for brightness and darkness balance, real detail restoration and boundary smoothness. It is transmitted as standard input to the adaptive recognition module for subsequent topology analysis and recognition model loading.

[0101] Example explanation:

[0102] Assumption:

[0103] Given the original image brightness I(x,y) = 210, atmospheric light A = 255, estimated distance d(x,y) = 30m, and the optimized atmospheric scattering coefficient β = 0.08, then:

[0104]

[0105] This module demonstrates that it can effectively restore physical brightness in low-transmittance environments, ensuring the image quality for subsequent recognition.

[0106] The adaptive recognition module includes:

[0107] 1. Receive illumination restoration image: First, receive the illumination restoration image I output from the physical illumination restoration module. fused (x,y) This image has the characteristic that the physical details of overexposed / underexposed areas are effectively compensated, which can improve the accuracy of subsequent topology recognition.

[0108] 2. Road Structure Topology Analysis and Lane Extraction: Lane extraction algorithms based on edge enhancement and semantic guidance (such as SCNN or ENet-Lane) are used to perform structure recognition on lane lines in the image, obtaining the centerline equation of each lane line. Lane centerlines are typically modeled using a two-dimensional polynomial approximation, represented as:

[0109] y = f(x) = a3x 3 +a2x 2 +a1x+a0;

[0110] By applying the first and second derivatives to the centerline, the rate of change of lane curvature k′ per unit path length is calculated:

[0111]

[0112] Where L is the unit path length (take L = 1m), and f′(x) and f″(x) are the first and second derivatives, respectively.

[0113] 3. Traffic sign spatial distribution density calculation: The pre-trained traffic sign detection model (YOLOv5-TrafficSign) is used to detect traffic signs in the image and generate bounding boxes for all traffic signs. The image is calibrated to a spatial region of real size according to the real-scene projection transformation, and at each 10m... 2 The number of traffic signs in the block The spatial distribution density of traffic signs is defined as follows:

[0114]

[0115] Where M is the total number of regions divided, and N is... i Let be the number of traffic signs in the i-th region.

[0116] 4. Topological Complexity Calculation and Model Selection: Based on the two key feature indicators extracted: lane curvature change rate κ′ and traffic sign spatial distribution density ρ sign The topological complexity C of the scenario is calculated using a linear weighted formula. topo The calculation is as follows:

[0117] C topo =0.7·κ′+0.3·ρ sign ;

[0118] Compare with a preset threshold θ (e.g., θ = 0.5):

[0119] If C topo If θ < , then load the lightweight recognition model;

[0120] If C topo If ≥θ, then load the high-precision recognition model;

[0121] In this invention, the lightweight recognition model adopts the MobileNet-SSD (Single Shot MultiBoxDetector) structure to quickly perform object detection and basic semantic segmentation tasks when the scene topological complexity is low, including:

[0122] (1) Model structure:

[0123] (1.1) Backbone Network: MobileNetV2 is used as the backbone feature extraction network, featuring a depthwise separable convolutional structure, which significantly reduces the number of parameters and computational complexity. Its basic module is the inverted residual block:

[0124]

[0125] (1.2) Detection Head: Convolutional layers are added to the output layers at multiple scales to output target prediction boxes (anchor boxes) of different sizes. Each anchor box simultaneously predicts the class probability and the bounding box offset.

[0126] (1.3) Semantic segmentation module (additional branch): A set of low-resolution decoding modules is added after the main feature map to upsample the feature map with C channels to the original size and perform pixel-wise softmax classification.

[0127] (2) Task execution:

[0128] Object detection: Using the SSD multi-scale detection head, a large number of candidate bounding boxes are generated in the image, and non-maximum suppression (NMS) is used to filter and output the final detected objects, such as pedestrians, vehicles, traffic lights, etc.

[0129] Semantic segmentation: Classifies each pixel of the image and outputs a coarse-level semantic region mask, which is used to divide the basic scene structure such as drivable area, background, and boundaries.

[0130] (3) Applicable scenarios: It is suitable for scenarios with straight lane lines, sparse traffic signs, and stable ambient lighting. It can complete basic environmental understanding tasks while ensuring recognition speed.

[0131] In this invention, the high-precision recognition model adopts a combined architecture of YOLOv7 and DeepLabv3+, which respectively handles object detection and semantic segmentation tasks, providing high-resolution and robust recognition results in high-topological-complexity scenarios. Specifically:

[0132] (1) Model structure:

[0133] (1.1) Object detection network (YOLOv7):

[0134] Backbone: Employs an enhanced CSPDarkNet model, combined with variable receptive field convolution (ELANmodule) for deep feature extraction;

[0135] Neck structure: Path Aggregation Network (PANet) is used for multi-scale feature fusion;

[0136] Detection Head: Outputs class confidence, bounding box regression value, and confidence score on three scales, suitable for scenes with dense small objects (such as areas with clustered traffic signs).

[0137] (1.2) Semantic Segmentation Network (DeepLabv3+):

[0138] Encoder: The backbone uses Xception or ResNet101 and is equipped with a Spatial Pyramid Pooling (ASPP) module to extract multi-scale contextual semantic features;

[0139] Decoder: It fuses high-level semantic features with shallow edge features, performs pixel-by-pixel classification, and outputs a high-resolution semantic label map.

[0140] (2) Task execution:

[0141] Target detection: Input the illumination-restored image into YOLOv7, predict the location and category of traffic participants through multi-layer features, including deformable targets, distant small targets, etc., and use an anchor-free decoding head to improve localization accuracy.

[0142] Semantic segmentation: DeepLabv3+ receives image input and outputs pixel-level semantic maps, distinguishing multiple structures such as lane line regions, drivable regions, obstacle regions, and green belts, ensuring high spatial topology restoration.

[0143] (3) Applicable scenarios: It is suitable for complex intersections, curves, high-speed merging, low light or severe occlusion scenarios, and can provide perception output with higher accuracy and boundary precision.

[0144] 5. Recognition Model Input and Target Recognition: Reconstructing the Illumination Image I fused (x, y) Input the selected model, perform object detection and semantic segmentation tasks, and output:

[0145] A set of object detection bounding boxes (such as pedestrians, vehicles, traffic lights);

[0146] Pixel-level semantic segmentation maps (such as road regions and obstacle boundaries);

[0147] All recognition results constitute the scene topology adaptive recognition result R. scene The data is then output in a unified format to the decision fusion module for multi-source fusion and trajectory correlation analysis.

[0148] Example explanation:

[0149] Assumption: The equation of the lane centerline is fitted as f(x) = 0.005x 2 +0.1x+2, within the path segment [0,1]m, κ′=0.75 is calculated; within 5 10m intervals... 2 If a total of 10 traffic signs are detected in the block, then:

[0150]

[0151] C topo =0.7·0.75 + 0.3·0.2 = 0.585;

[0152] Because of C topo If θ = 0.5, the system will load a high-precision recognition model to perform the target detection task.

[0153] The decision fusion module includes:

[0154] 1. Acquisition of LiDAR target trajectory prediction data:

[0155] (1.1) Acquisition and preprocessing of raw point cloud data from lidar: Raw point cloud data is acquired using a high-frequency scanning 3D lidar module. Each point p i =(x i ,y i ,z i Voxel grid filters are used for downsampling to reduce the amount of data while preserving local geometry.

[0156] (1.2) Euclidean Clustering Segmentation for Extracting Independent Moving Target Point Cloud Clusters: Perform Euclidean distance-based clustering segmentation on the denoised point cloud to extract multiple moving target point cloud clusters. Each point cloud cluster represents an independent target.

[0157] (1.3) Multi-target tracking trajectory construction: Data association is performed on each point cloud cluster within 5 consecutive frames, and a stable ID-tracking pair is constructed using the Hungarian algorithm to obtain the trajectory sequence of the target from time t-4 to time t:

[0158]

[0159] (1.4) Kalman filter predictor estimates motion state: For each trajectory input to the Kalman filter model, the motion state in the current frame is estimated:

[0160]

[0161] Where v is the velocity component and a is the acceleration component, the predicted trajectory point sequence within the future Δt = 100ms is:

[0162]

[0163] Sequence of all target trajectory points Integrate and generate LiDAR target trajectory prediction data.

[0164] 2. Cross-modal association matching: The visual target bounding boxes B = {b1, b2, ..., b...} from the adaptive recognition module are matched. m} and the aforementioned radar trajectory data Matching is performed. First, the 3D trajectory is projected onto the image coordinate system from the camera's perspective (spatial mapping is completed using the calibrated extrinsic parameter matrix P), and the bounding box b for each pair of visual targets is calculated. i With trajectory point set The cost of the center point distance D ij :

[0165]

[0166] Using the Hungarian algorithm on the cost matrix D∈R m×k Perform a minimum match to generate related matching pairs.

[0167] 3. Spatiotemporal consistency verification and identification of isolated false detection targets:

[0168] (3.1) Spatiotemporal consistency check: For each matching pair Calculate the position offset δ within 5 consecutive frames. pos Angle δ of deviation from the direction of motion θ ;

[0169] Position offset:

[0170] Angle of deviation in direction of motion:

[0171] in These are the visual and radar trajectory direction vectors, respectively.

[0172] (3.2) Screening out isolated false positives:

[0173] If a matching pair satisfies any of the following conditions:

[0174] δ pos >1.5m or δ θ >15°;

[0175] Then add the visual target to the set of non-physically feasible targets.

[0176] 4. Final Recognition Result Generation and Output: This involves generating and outputting all elements contained in the scene topology adaptive recognition result. Target filtering and elimination generate the final recognition result set. It is then output to the vehicle control system for path planning and obstacle avoidance decisions.

[0177] Example explanation:

[0178] Assuming that in the current frame, the visual recognition box b1 matches the radar target T1, the maximum offset over 5 consecutive frames is 1.8m, and the motion direction deviation is 10°, then according to the strategy:

[0179] Although the angle difference is within the range, the position offset exceeds the threshold of 1.5m. Therefore, b1 is identified as an isolated false detection target and is removed.

[0180] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0181] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An in-vehicle AI visual recognition terminal for intelligent driving scenarios, characterized in that, It includes a spatiotemporal alignment module, a physical illumination restoration module, an adaptive recognition module, and a decision fusion module, among which; The spatiotemporal alignment module is used to acquire the original image from the camera and the point cloud data from the millimeter-wave radar, generate the optical flow field of the moving target based on the radar point cloud data, compensate for the sampling time difference between the camera and the radar through back projection of the flow field, and output the spatiotemporal aligned image. The physical illumination restoration module inputs the spatiotemporally aligned image into the physically parameterized illumination model. The physically parameterized illumination model integrates the atmospheric scattering equation and the camera response function, and reconstructs the physical real details of the overexposed / underexposed areas by solving the illumination transmission equation, and outputs the illumination restoration image. The adaptive recognition module performs road structure topology analysis on the illumination restoration image, extracts the lane line curvature change rate and traffic sign spatial distribution density, dynamically loads a lightweight recognition model or a high-precision recognition model according to the topology complexity, and outputs scene topology adaptive recognition results. The decision fusion module performs cross-modal correlation between the scene topology adaptive recognition result and the LiDAR target trajectory prediction, filters out isolated false detection targets through spatiotemporal consistency verification, and outputs the final recognition result.

2. The in-vehicle AI visual recognition terminal for intelligent driving scenarios according to claim 1, characterized in that, The spatiotemporal alignment module includes: Simultaneously, it acquires raw images from the camera and point cloud data from millimeter-wave radar; Moving target clustering and segmentation are performed on millimeter-wave radar point cloud data, Doppler velocity vector of each moving target is extracted, and Doppler velocity vectors of all moving targets are superimposed to generate dense moving target optical flow field; The dynamic interpolation weights of the optical flow field of the moving target are calculated based on the difference between the camera exposure timestamp and the millimeter-wave radar scanning period. The optical flow field of the moving target is back-projected onto the original image of the camera according to the dynamic interpolation weight, and the original image of the camera is remapped through bilinear sampling; The original image from the remapped camera is processed for edge smoothing to generate a spatiotemporally aligned image, which is then output to the physical illumination restoration module.

3. The in-vehicle AI visual recognition terminal for intelligent driving scenarios according to claim 2, characterized in that, The physical illumination restoration module includes: Receive the spatiotemporal alignment image output by the spatiotemporal alignment module; The spatiotemporally aligned image is input into the physically parameterized lighting model, which is constructed by fusing the atmospheric scattering equation and the camera response function; The atmospheric scattering coefficient of the scene is calculated using the atmospheric scattering equation, and the atmospheric scattering coefficient is adaptively optimized based on the sky region segmentation results. Nonlinear response correction is performed on the spatiotemporally aligned image using the camera response function to generate a sensor characteristic calibration image.

4. The in-vehicle AI visual recognition terminal for intelligent driving scenarios according to claim 3, characterized in that, The physical illumination restoration module also includes: By substituting the atmospheric scattering coefficient and sensor characteristics into the irradiance transfer equation, the Monte Carlo ray tracing algorithm is used to solve the radiative transfer in overexposed and underexposed areas, and the physical real details are reconstructed. The reconstructed image is subjected to gradient domain fusion processing to generate an illumination restoration image, which is then output to the adaptive recognition module.

5. The in-vehicle AI visual recognition terminal for intelligent driving scenarios according to claim 4, characterized in that, The adaptive recognition module includes: Receive the illumination restoration image output by the physical illumination restoration module; Perform road structure topology analysis on the illumination restoration image to extract the lane centerline equation; Calculate the rate of change of lane curvature per meter of path length based on the lane centerline equation; Detect bounding boxes of traffic signs in illuminated restored images and calculate the spatial distribution density of traffic signs per unit area; Calculate the topological complexity based on the rate of change of lane curvature and the spatial distribution density of traffic signs: Topological complexity = 0.7 × lane line curvature change rate + 0.3 × traffic sign spatial distribution density.

6. The in-vehicle AI visual recognition terminal for intelligent driving scenarios according to claim 5, characterized in that, The process of dynamically loading a lightweight or high-precision recognition model based on topological complexity and outputting scene topology adaptive recognition results includes: If the topology complexity is less than the threshold, a lightweight recognition model is loaded; if the topology complexity is greater than or equal to the threshold, a high-precision recognition model is loaded. The illumination-restored image is input into the loaded recognition model for target detection and semantic segmentation, generating scene topology adaptive recognition results and outputting them to the decision fusion module.

7. The in-vehicle AI visual recognition terminal for intelligent driving scenarios according to claim 6, characterized in that, The decision fusion module includes: Receive the scene topology adaptive recognition result output by the adaptive recognition module, and simultaneously acquire the target trajectory prediction data of the lidar; The scene topology adaptive recognition results are cross-modal correlated with the LiDAR target trajectory prediction data, and the visual target box and LiDAR trajectory points are matched by the Hungarian algorithm to generate correlation matching pairs. Spatiotemporal consistency verification is performed on the associated matching pairs, and the position offset and motion direction deviation of the visual target and the lidar trajectory are calculated within 5 consecutive frames.

8. The in-vehicle AI visual recognition terminal for intelligent driving scenarios according to claim 7, characterized in that, The decision fusion module also includes: If the position offset is greater than 1.5 meters or the movement direction deviation is greater than 15 degrees, the target is determined to be an isolated false detection target and a set of non-physically feasible targets is generated. The system filters out targets not included in the physically feasible target set from the scene topology adaptive recognition results, generates the final recognition result, and outputs it to the vehicle control system.

9. The in-vehicle AI visual recognition terminal for intelligent driving scenarios according to claim 8, characterized in that, The acquisition of lidar target trajectory prediction data includes: Collect raw point cloud data from lidar and perform point cloud denoising through voxel grid downsampling; Euclidean clustering was performed on the denoised raw point cloud data of the lidar to separate independent moving target point cloud clusters. Perform multi-target tracking on each moving target point cloud cluster for 5 consecutive frames to generate multi-target tracking trajectory; Based on the multi-target tracking trajectory, the motion state of each target is estimated by a Kalman filter predictor, and the motion state includes position coordinates, velocity vector and acceleration vector; Establish a kinematic model of the target based on its motion state, and predict the target trajectory point sequence within the next 100ms. By integrating the trajectory point sequences of all targets, target trajectory prediction data for lidar is generated.

Images