A vehicle monitoring system based on deep learning

Abstract

The present invention discloses a vehicle-mounted monitoring system based on deep learning, which specifically relates to the field of vehicle-mounted multimodal collaborative perception technology. It is used to solve the problems of temporal and spatial gradual deviation accumulation and distortion of target state estimation in high-speed scenes caused by hardware synchronization limitations in the existing fusion of vision and millimeter-wave radar; it realizes high-precision alignment and adaptive correction of multimodal data through dynamic spatiotemporal baseline compensation and closed-loop feedback optimization mechanism; it generates dynamic compensation coefficients based on the prediction of sensor spatiotemporal distribution patterns, and constructs a nonlinear correlation model of multimodal data streams to eliminate sampling frequency differences and transmission fluctuation interference; it separates visual motion artifacts and radar positioning jump noise through cross-modal feature decoupling technology, and extracts high-confidence feature fragments by combining historical trajectory screening and real-time residual analysis; it establishes a closed-loop architecture of fusion weight allocation and reverse correction of compensation parameters to realize dynamic calibration and error tracing of multi-source data collaborative perception.

Description

Technical Field

[0001] The present invention relates to the field of vehicle-mounted multimodal collaborative perception technology, and more specifically, to a vehicle-mounted monitoring system based on deep learning. Background Art

[0002] The rapid development of intelligent driving technology has led to the gradual adoption of a multi-modal sensor collaborative architecture for vehicle-mounted environment perception to enhance target detection and trajectory prediction capabilities in complex scenarios. The fusion of visual sensors and millimeter-wave radars has become the mainstream technology due to their complementary characteristics. Visual sensors can provide rich texture information, while millimeter-wave radars have precise speed measurement and anti-interference capabilities. Together, these two can effectively cope with lighting changes, occlusions, and other situations. Existing technologies typically align and fuse multimodal data based on the assumption of time synchronization, relying on hardware clock synchronization or fixed delay compensation strategies to achieve spatiotemporal correlation of sensor data.

[0003] However, in practical applications, due to differences in hardware acquisition frequencies among different sensors and dynamic fluctuations in data transmission links, it is difficult to maintain strict alignment of timestamps in multimodal data streams, resulting in progressive deviations in the spatiotemporal feature fusion process. Especially in high-speed vehicle maneuvering scenarios, progressive deviations will be nonlinearly amplified with increasing perception distance, causing distortion in the estimation of the target's motion state, seriously restricting the reliability and safety of decision-making in critical scenarios based on vehicle-mounted environment perception. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a vehicle monitoring system based on deep learning to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A vehicle monitoring system based on deep learning, comprising:

[0007] Spatiotemporal prediction module: collects visual data and millimeter-wave radar data, and predicts spatiotemporal distribution patterns based on motion parameters in the millimeter-wave radar data;

[0008] Compensation calculation module: dynamically generates spatiotemporal baseline compensation coefficients for visual data and millimeter-wave radar data based on spatiotemporal distribution patterns;

[0009] Correlation Verification Module: Builds a spatiotemporal correlation model of multimodal data streams based on spatiotemporal baseline compensation coefficients to identify and verify sensitive areas in high-speed scenarios;

[0010] Feature screening module: For verified sensitive areas, pseudo-sensitive areas are eliminated based on historical trajectory data, and high-confidence feature segments are screened based on the correlation between visual features and millimeter-wave radar velocity distribution;

[0011] Decoupling correction module: decouples high-confidence feature segments, suppresses visual motion artifacts, corrects spatial positioning jumps in millimeter-wave radar data, and generates aligned multimodal feature vectors;

[0012] Closed-loop optimization module: The trajectory prediction model is constructed based on the dynamic allocation of fusion weights based on the multimodal feature vectors, and the spatiotemporal baseline compensation coefficient is reversely corrected through the residual error between the predicted trajectory and the measured trajectory.

[0013] In a preferred embodiment, verifying the sensitive area includes verifying the physical spatial consistency of the sensitive area by correlating characteristic mutations of visual data and millimeter-wave radar data.

[0014] In a preferred embodiment, collecting visual data and millimeter-wave radar data, and predicting a spatiotemporal distribution pattern based on motion parameters in the millimeter-wave radar data, includes:

[0015] Vision data and millimeter-wave radar data are collected and pre-processed synchronously through hardware synchronization trigger signals;

[0016] Decompose the motion parameters in millimeter-wave radar data into acceleration change rate and motion direction offset;

[0017] The phase offset in the time dimension is calculated based on the acceleration change rate, and the projection deviation in the space dimension is calculated based on the motion direction offset;

[0018] Perform time-frequency analysis on the phase offset and projection deviation to extract the high-energy fluctuation interval in the frequency domain characteristics;

[0019] According to the geometric projection relationship between the high-energy fluctuation range and the sensor installation location, a spatiotemporal distribution pattern including continuous error and mutation error classification is generated.

[0020] In a preferred embodiment, dynamically generating spatiotemporal baseline compensation coefficients for visual data and millimeter-wave radar data based on the spatiotemporal distribution pattern includes:

[0021] Calculating a time compensation component based on the phase offset in the time dimension, wherein the time compensation component is the product of the phase offset and the sampling interval;

[0022] Calculating a spatial compensation component based on the projection deviation of the spatial dimension, where the spatial compensation component is the ratio of the projection deviation to the sensor installation spacing;

[0023] The time compensation component and the space compensation component are linearly weighted fused to generate the spatiotemporal baseline compensation coefficient. The weight of the linear weighted fusion is dynamically allocated according to the ratio of continuous error to sudden error in the spatiotemporal distribution pattern.

[0024] In a preferred embodiment, a spatiotemporal correlation model of a multimodal data stream is constructed based on the spatiotemporal baseline compensation coefficient to identify sensitive areas in high-speed scenarios, including:

[0025] Adjust the time interpolation step and spatial coordinate transformation parameters of the multimodal data stream based on the spatiotemporal baseline compensation coefficient;

[0026] Perform time-domain filtering on the velocity mutation features in millimeter-wave radar data to extract candidate sensitive areas where the velocity change rate exceeds the dynamic threshold;

[0027] The radar coordinates of the candidate sensitive area are mapped to the visual sensor image plane through the spatial coordinate transformation parameters to generate the corresponding pixel coordinate area;

[0028] The edge gradient mutation features of the mapped pixel area in the visual data are extracted, and the area where the edge gradient amplitude exceeds the preset gradient threshold is marked as a sensitive area.

[0029] In a preferred embodiment, the time interpolation step is generated by the time compensation component and the preset interpolation window length, and the spatial coordinate transformation parameter is generated by associating the spatial compensation component with the visual sensor resolution.

[0030] In a preferred embodiment, the physical spatial consistency of the sensitive area is verified by correlating the characteristic mutations of the visual data and the millimeter-wave radar data, specifically:

[0031] Count the number of pixels whose edge gradient amplitude exceeds the preset gradient threshold;

[0032] Calculate the spatial distribution matching degree between the velocity mutation amplitude in the radar sensitive area and the number of edge gradient pixels in the visual sensitive area;

[0033] If the spatial distribution matching degree exceeds the preset matching threshold and the overlap between the pixel coordinate area and the visual sensitive area exceeds the preset ratio, it is judged that the physical space is consistent and the sensitive area verification passes; otherwise, it is marked as an invalid area and eliminated, and the sensitive area verification fails.

[0034] In a preferred embodiment, for verified sensitive areas, pseudo-sensitive areas are eliminated based on historical trajectory data, and high-confidence feature segments are screened based on the correlation between visual features and millimeter-wave radar velocity distribution, including:

[0035] Based on the continuity of the target acceleration rate in the historical trajectory data, a physically feasible range of the acceleration rate is constructed;

[0036] Eliminate pseudo-sensitive areas where the acceleration change rate exceeds the physically feasible range. At the same time, perform multi-frame trajectory smoothness verification on sensitive areas where the acceleration change rate is at the boundary of the range. If the number of sudden changes in acceleration direction within several consecutive frames exceeds the preset number, it is determined to be a pseudo area.

[0037] Extract visual feature segments from verified sensitive areas, calculate the spatial covariance between the corresponding edge gradient directional distribution and the millimeter-wave radar velocity directional distribution, and extract the covariance benchmark value for similar scenarios in historical trajectories.

[0038] The fusion weight is dynamically assigned based on the deviation between the current spatial covariance and the covariance baseline value. The smaller the deviation, the higher the visual feature weight, and the larger the deviation, the higher the millimeter wave radar speed weight.

[0039] The spatial and temporal features of vision and radar in the multimodal feature segments are corrected bidirectionally. The visual features correct the spatial positioning jump of the radar velocity distribution, and the radar velocity corrects the motion blur artifacts of the visual features, thus generating high-confidence feature segments that are consistent in time and space.

[0040] In a preferred embodiment, high-confidence feature segments are decoupled to suppress visual motion artifacts and correct spatial positioning jumps of millimeter-wave radar data to generate aligned multimodal feature vectors, including:

[0041] Decompose high-confidence feature segments based on the spatiotemporal dimension to separate the motion artifact components in visual features from the positioning jump components in millimeter-wave radar features;

[0042] Perform gradient direction filtering on motion artifact components to suppress gradient components that deviate from the radar velocity distribution direction by more than a preset angle;

[0043] The positioning jump components are interpolated spatially and continuously, and the interpolation step size is dynamically adjusted according to the consistency of target displacement in adjacent frames in the visual features;

[0044] The filtered visual features and the interpolated radar features are weightedly concatenated according to the spatiotemporal baseline compensation coefficient to generate a channel-aligned multimodal feature vector.

[0045] Based on the channel weight distribution of similar scenes in historical trajectories, the multimodal feature vector is normalized and energy calibrated.

[0046] In a preferred embodiment, a trajectory prediction model is constructed based on the dynamic allocation of fusion weights of multimodal feature vectors, and the spatiotemporal baseline compensation coefficient is reversely corrected by the residual error between the predicted trajectory and the measured trajectory, including:

[0047] Based on the channel weight distribution of historical multimodal feature vectors, the trajectory prediction model is constructed by dynamically allocating fusion weights. The weight allocation rule is generated based on the historical scene clustering results and the current feature deviation.

[0048] Extract the residual error between the predicted trajectory and the measured trajectory. The residual error includes the horizontal displacement deviation, direction angle deviation and velocity change rate difference.

[0049] The spatial and temporal covariance analysis of the residual error is performed to separate the systematic error components caused by the deviation of the spatial and temporal baseline compensation coefficient.

[0050] According to the amplitude direction of the systematic error component, the time compensation component and the space compensation component in the spatiotemporal baseline compensation coefficient are reversely adjusted;

[0051] The corrected spatiotemporal baseline compensation coefficients are iteratively input into the spatiotemporal correlation model to update the multimodal data alignment parameters.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. This invention significantly improves the spatiotemporal feature alignment accuracy and dynamic scene adaptability of the vehicle-mounted environmental perception system through a dynamic spatiotemporal baseline compensation mechanism and closed-loop feedback optimization architecture for multimodal data streams. Based on the autonomous modeling and compensation coefficient generation method for the spatiotemporal distribution pattern of sensor data, it achieves high-precision dynamic spatiotemporal correlation between visual data and millimeter-wave radar data. By constructing a nonlinear compensation model for multimodal data streams, it effectively eliminates the progressive spatiotemporal deviation accumulation caused by sensor sampling frequency differences and transmission delays in high-speed maneuvering scenarios, significantly reducing the risk of distortion in the motion state estimation of distant targets and enhancing the robustness of perception decisions under complex road conditions.

[0054] 2. Enhance the collaborative perception quality of multi-source heterogeneous data through cross-modal feature decoupling, correction, and self-optimization fusion mechanisms. A two-way screening strategy based on physical constraints and historical trajectory features accurately separates visual motion artifacts from radar positioning jump interference, enabling rapid extraction of high-confidence feature fragments. A closed-loop compensation parameter iteration system integrating residual feedback can trace sensor errors in real time in dynamic scenarios and autonomously adjust spatiotemporal alignment parameters to ensure the long-term stability of multimodal data fusion. This allows the system to maintain accurate expression and efficient collaboration of multi-dimensional spatiotemporal features under extreme lighting, target occlusion, and sudden maneuvers, providing low-latency, highly reliable environmental perception input for the intelligent driving control module, enhancing safety redundancy and decision-making fault tolerance in complex traffic scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a structural diagram of a vehicle monitoring system based on deep learning in the present invention;

[0056] Figure 2 This is a flow chart for verifying sensitive areas in the present invention. DETAILED DESCRIPTION

[0057] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] Example: Figure 1 A schematic diagram of the structure of a vehicle-mounted monitoring system based on deep learning is provided in the present invention. The vehicle-mounted monitoring system based on deep learning includes:

[0059] Spatiotemporal prediction module: collects visual data and millimeter-wave radar data, and predicts spatiotemporal distribution patterns based on motion parameters in the millimeter-wave radar data;

[0060] Compensation calculation module: dynamically generates spatiotemporal baseline compensation coefficients for visual data and millimeter-wave radar data based on spatiotemporal distribution patterns;

[0061] Correlation Verification Module: Builds a spatiotemporal correlation model of multimodal data streams based on spatiotemporal baseline compensation coefficients to identify and verify sensitive areas in high-speed scenarios;

[0062] Feature screening module: For verified sensitive areas, pseudo-sensitive areas are eliminated based on historical trajectory data, and high-confidence feature segments are screened based on the correlation between visual features and millimeter-wave radar velocity distribution;

[0063] Decoupling correction module: decouples high-confidence feature segments, suppresses visual motion artifacts, corrects spatial positioning jumps in millimeter-wave radar data, and generates aligned multimodal feature vectors;

[0064] Closed-loop optimization module: The trajectory prediction model is constructed based on the dynamic allocation of fusion weights based on the multimodal feature vectors, and the spatiotemporal baseline compensation coefficient is reversely corrected through the residual error between the predicted trajectory and the measured trajectory.

[0065] Collect visual data and millimeter-wave radar data, and predict spatiotemporal distribution patterns based on motion parameters in the millimeter-wave radar data, including:

[0066] Vision data and millimeter-wave radar data are collected and pre-processed synchronously through hardware synchronization trigger signals;

[0067] Decompose the motion parameters in millimeter-wave radar data into acceleration change rate and motion direction offset;

[0068] The phase offset in the time dimension is calculated based on the acceleration change rate, and the projection deviation in the space dimension is calculated based on the motion direction offset;

[0069] Perform time-frequency analysis on the phase offset and projection deviation to extract the high-energy fluctuation interval in the frequency domain characteristics;

[0070] According to the geometric projection relationship between the high-energy fluctuation range and the sensor installation location, a spatiotemporal distribution pattern including continuous error and mutation error classification is generated.

[0071] A hardware synchronization trigger signal is generated by the onboard central controller and sent simultaneously to the visual sensor and millimeter-wave radar via a physical level signal, ensuring that the start time deviation of data acquisition is less than 1 millisecond. The visual sensor collects image data at a fixed frequency, such as 30Hz, while the millimeter-wave radar collects target motion parameters at a higher frequency, such as a 50Hz pulse repetition frequency. Median filtering is performed on the visual data to eliminate sudden changes in illumination noise. For example, a 3×3 pixel window is used to sort neighboring pixels in the image and replace the center pixel value with the median value. Doppler shift compensation is performed on the millimeter-wave radar data. For example, the velocity measurement is corrected based on the linear relationship between the target radial velocity and the radar wave frequency to eliminate velocity estimation bias of moving targets.

[0072] The acceleration rate is obtained by calculating the second-order derivative of velocity data at adjacent time points, for example, calculating and differentiating the velocity change every 0.02 seconds. The motion direction offset is calculated by taking the sine difference between the target azimuth angles of the current frame and the previous frame, for example, by mapping the azimuth angle difference to a lateral displacement on a two-dimensional plane. The phase offset in the time dimension is calculated based on the acceleration rate. Specifically, the acceleration rate is multiplied by a preset time window length to obtain the cumulative phase deviation on the time axis. The preset time window length is set based on the vehicle's maximum expected acceleration, for example, 0.2 seconds for a typical passenger car. The spatial projection offset is calculated based on the motion direction offset. Specifically, the directional offset is converted into a pixel displacement in the visual sensor coordinate system. Coordinate transformation is performed based on the geometric projection relationship between the millimeter-wave radar and visual sensor installation locations. This geometric projection relationship is pre-established through calibration experiments. For example, while the vehicle is stationary, visual images of a calibration plate and radar point cloud data are collected, and the affine transformation matrix is ​​calculated using the least squares method.

[0073] A time-frequency analysis is performed on the phase offset and projection deviation, for example, by using a short-time Fourier transform (SFT) to convert the time-domain signal into a frequency-domain energy distribution spectrum. The SFT window function is set to 0.1 seconds, with an overlap rate of 50% and a frequency resolution of 10 Hz.

[0074] High-energy fluctuation intervals in the frequency domain features, where the energy value exceeds a preset threshold, are extracted. For example, frequency bands whose energy value exceeds a historical data statistical threshold are marked as high-energy intervals. The preset threshold is dynamically adjusted based on the noise level in normal vehicle driving scenarios, for example, the top 10% percentile of the frequency domain energy distribution is used as the decision boundary. Based on the geometric projection relationship between the high-energy fluctuation interval and the sensor installation location, a spatiotemporal distribution pattern is generated, which includes continuous error and sudden error classification. Continuous error corresponds to the smooth fluctuation of the phase offset in uniform speed scenarios, such as the phase offset fluctuating within ±5 milliseconds when the vehicle is moving straight at a constant speed. Sudden error corresponds to the step-like jump of the projection deviation in emergency acceleration or steering scenarios, such as the projection deviation increasing by more than 20 pixels within 0.1 second during sudden braking.

[0075] The hardware synchronization trigger signal can be a GPIO level signal, a CAN bus trigger command, or a PPS synchronization pulse. For example, when using GPIO trigger, the controller outputs a 5V high level pulse to the sensor trigger pin. Doppler shift compensation is based on the formula Corrected target radial velocity, where represents the radial velocity of the target object (i.e., the moving speed of the target relative to the millimeter-wave radar), is the echo frequency offset, is the speed of light, is the radar transmission frequency. For example, the frequency shift compensation range for a 77GHz millimeter-wave radar is set to ±500Hz. The preset time window length is shortened to 0.1 seconds when the vehicle's maximum acceleration is 10m / s². The frequency domain energy distribution spectrum extracts the energy peak interval within the 0-100Hz frequency band. The affine transformation matrix calibration error is less than 0.5 pixels, for example, by obtaining a one-to-one correspondence between the corner coordinates of a checkerboard calibration plate and the radar's 3D coordinates.

[0076] Hardware synchronization triggering ensures data time alignment, providing time-consistent input for subsequent parameter decomposition. Decomposition of the acceleration rate and directional offset breaks down motion parameters into error sources in the temporal and spatial dimensions. Time-frequency analysis focuses on high-energy regions in the frequency domain, reducing the impact of noise interference on error classification. Geometric projection maps radar data to the visual coordinate system, ensuring physical consistency in spatial dimension calculations. The classification results of continuous and sudden errors provide a priori basis for the generation of dynamic compensation coefficients. For example, continuous errors are compensated using linear interpolation, while sudden errors are compensated using nonlinear filtering.

[0077] Dynamically generate spatiotemporal baseline compensation coefficients for visual data and millimeter-wave radar data based on spatiotemporal distribution patterns, including:

[0078] Calculating a time compensation component based on the phase offset in the time dimension, wherein the time compensation component is the product of the phase offset and the sampling interval;

[0079] Calculating a spatial compensation component based on the projection deviation of the spatial dimension, where the spatial compensation component is the ratio of the projection deviation to the sensor installation spacing;

[0080] The time compensation component and the space compensation component are linearly weighted fused to generate the spatiotemporal baseline compensation coefficient. The weight of the linear weighted fusion is dynamically allocated according to the ratio of continuous error to sudden error in the spatiotemporal distribution pattern.

[0081] The spatiotemporal distribution pattern includes the classification results of continuous error and mutation error, among which the continuous error is the steady fluctuation of the phase offset in the time dimension, and the mutation error is the step-like jump of the projection deviation in the spatial dimension.

[0082] The time compensation component is calculated as the product of the phase offset and the sampling interval. The phase offset is calculated by multiplying the acceleration rate by the length of the preset time window. For example, if the acceleration rate is 2 m / s² and the time window is 0.2 seconds, the phase offset is 0.4 seconds. The sampling interval is the difference between the acquisition periods of the vision sensor and the millimeter-wave radar. For example, if the acquisition period of the vision sensor is 33 milliseconds and the acquisition period of the millimeter-wave radar is 20 milliseconds, the sampling interval is 13 milliseconds. The time compensation component quantifies the cumulative effect of timestamp alignment misalignment. For example, the product of a phase offset of 0.4 seconds and a sampling interval of 13 milliseconds is 5.2 milliseconds².

[0083] The spatial compensation component is calculated as the ratio of the projected deviation to the sensor mounting spacing. The projected deviation is calculated by mapping the motion direction offset to the pixel displacement in the vision sensor coordinate system. For example, a 3-degree directional offset results in a projected deviation of 60 pixels.

[0084] The sensor spacing is the physical distance between the vision sensor and the millimeter-wave radar, measured through calibration experiments. For example, it can be measured and recorded as 0.5 meters using a laser rangefinder. The spatial compensation component quantifies the scaling ratio of spatial coordinate deviation. For example, the ratio of 60 pixels of projection deviation to a 0.5-meter spacing is 120 pixels / meter.

[0085] The linearly weighted fusion weights of the temporal and spatial compensation components are dynamically allocated based on the ratio of continuous to sudden errors. The error ratio is calculated by counting the proportion of the two types of errors in the spatiotemporal distribution. For example, if there are 80 continuous errors and 20 sudden errors, the proportion of continuous errors is 80%. When the proportion of continuous errors exceeds 70%, the weight of the temporal compensation component is set to 0.8, and the weight of the spatial compensation component is set to 0.2. When the proportion of sudden errors exceeds 60%, the weight of the temporal compensation component is set to 0.3, and the weight of the spatial compensation component is set to 0.7. The weight allocation strategy is based on the statistical patterns of historical driving scenario data. For example, the proportion of continuous errors in highway scenarios is generally higher than 70%, and the proportion of sudden errors in urban congestion scenarios is generally higher than 60%.

[0086] The spatiotemporal baseline compensation coefficient is generated using a weighted fusion formula. For example, if the temporal compensation component is 5.2 milliseconds² and the spatial compensation component is 120 pixels / meter, with weights of 0.8 and 0.2, respectively, the spatiotemporal baseline compensation coefficient is 5.2 × 0.8 + 120 × 0.2 = 28.16. This coefficient is input into the spatiotemporal correlation model to adjust the model's temporal interpolation step size and spatial coordinate transformation parameters. For example, the temporal interpolation step size can be adjusted from the default 10 milliseconds to a dynamic value corresponding to an interpolation interval of 28.16 milliseconds², while the spatial coordinate transformation parameters adjust the pixel mapping ratio of the target in the visual image to 120 pixels / meter.

[0087] Among them, before calculating the spatiotemporal baseline compensation coefficient, the time compensation component and the space compensation component need to be dimensionless.

[0088] A spatiotemporal correlation model of multimodal data streams is constructed based on spatiotemporal baseline compensation coefficients to identify sensitive areas in high-speed scenarios, including:

[0089] Adjust the time interpolation step and spatial coordinate transformation parameters of the multimodal data stream based on the spatiotemporal baseline compensation coefficient. The time interpolation step is generated by the time compensation component and the preset interpolation window length, and the spatial coordinate transformation parameters are generated by associating the spatial compensation component with the visual sensor resolution.

[0090] Perform time-domain filtering on the velocity mutation features in millimeter-wave radar data to extract candidate sensitive areas where the velocity change rate exceeds the dynamic threshold;

[0091] The radar coordinates of the candidate sensitive area are mapped to the visual sensor image plane through the spatial coordinate transformation parameters to generate the corresponding pixel coordinate area;

[0092] The edge gradient mutation features of the mapped pixel area in the visual data are extracted, and the area where the edge gradient amplitude exceeds the preset gradient threshold is marked as a sensitive area.

[0093] The time compensation component of the spatiotemporal baseline compensation coefficient is calculated by multiplying the phase offset by the sampling interval. For example, when the phase offset is 0.4 seconds and the sampling interval is 13 milliseconds, the time compensation component is 5.2 milliseconds². The spatial compensation component is calculated by the ratio of the projection offset to the sensor installation spacing. For example, when the projection offset is 60 pixels and the installation spacing is 0.5 meters, the spatial compensation component is 120 pixels / meter.

[0094] The temporal interpolation step is calculated by multiplying the temporal compensation component by the preset interpolation window length. The preset interpolation window length is the same as the preset temporal window length used to calculate the phase offset. For example, when the preset temporal window length is 0.2 seconds, the temporal interpolation step is 5.2 milliseconds² × 0.2 seconds = 1.04 seconds. The spatial coordinate transformation parameter is calculated by the ratio of the spatial compensation component to the vision sensor resolution. For example, when the vision sensor resolution is 200 pixels / meter, the spatial coordinate transformation parameter is 120 pixels / meter ÷ 200 pixels / meter = 0.6.

[0095] Time-domain filtering is performed on the speed mutation features in millimeter-wave radar data to extract candidate sensitive areas where the speed change rate exceeds a dynamic threshold. The dynamic threshold is set based on the statistical distribution of speed mutation amplitudes in high-speed scenarios in historical data. Specifically, the top 5% percentile of speed mutation data on highways over the past month is used as the dynamic threshold.

[0096] Time-domain filtering uses a sliding window averaging method. The window length matches the preset time window length. For example, a 0.2-second window length is used to calculate the moving average of the velocity change rate. Areas exceeding the dynamic threshold are marked as candidate sensitive areas. For example, if the velocity change rate in an area increases from 20 m / s to 30 m / s within 0.2 seconds, the velocity change rate is 50 m / s², exceeding the dynamic threshold of 30 m / s² and thus marked as a candidate sensitive area.

[0097] The target coordinates of the candidate sensitive area in the millimeter-wave radar coordinate system are mapped to the visual sensor image plane using spatial coordinate transformation parameters to generate the corresponding pixel coordinate area. The spatial coordinate transformation parameter is calculated by the ratio of the spatial compensation component to the visual sensor resolution. For example, when the spatial compensation component is 120 pixels / meter and the visual sensor resolution is 200 pixels / meter, the coordinate transformation parameter is 0.6.

[0098] The target coordinates in the millimeter-wave radar coordinate system are converted to pixel coordinates on the image plane using an affine transformation matrix. This matrix is ​​pre-acquired through calibration experiments. For example, with the vehicle stationary, visual images of a checkerboard calibration plate and radar point cloud data are collected. The six parameters of the transformation matrix are calculated using the least squares method to ensure a coordinate mapping error of less than 0.5 pixels. For example, the target coordinates in the radar coordinate system (10 meters, 2 meters) are converted to image coordinates (10 × 0.6, 2 × 0.6) = (6 pixels, 1.2 pixels).

[0099] Edge gradient mutation features are extracted from the mapped pixel areas in the visual data, and regions whose edge gradient amplitude exceeds a preset gradient threshold are marked as sensitive regions. Edge gradient mutation features are detected using the Sobel operator, which uses a 3×3 convolution kernel to calculate the horizontal and vertical gradient amplitudes of the image region and merges them to form an edge gradient map. The preset gradient threshold is set based on the noise characteristics of the visual sensor, for example, 50 in low-light conditions and 30 in normal light conditions. The noise characteristics are measured through dark-field imaging experiments. For example, in a night scene, if the edge gradient amplitude of a region is 60, which exceeds the preset threshold of 50, it is marked as a candidate visually sensitive region. If the gradient amplitude of a region in a daytime scene is 25, which is below the threshold of 30, it is not marked as a sensitive region.

[0100] The preset gradient threshold is based on noise level measurements of the visual sensor during dark-field experiments. For example, 100 frames of imagery are collected in complete darkness, and the maximum noise gradient amplitude is used as a reference. The calibration error of the affine transformation matrix is ​​calculated using the residuals of the checkerboard corner coordinates. A residual mean value less than 0.5 pixels is considered valid.

[0101] Adjusting the temporal interpolation step size directly affects the temporal alignment accuracy of multimodal data streams. For example, a longer interpolation step size may lead to the accumulation of timestamp deviations. The generation of spatial coordinate transformation parameters depends on the spatial compensation component and the resolution of the visual sensor to ensure the physical consistency of radar data to visual coordinates. Candidate sensitive areas extracted by time-domain filtering are converted into pixel areas on the visual plane through coordinate mapping, providing a spatial positioning basis for subsequent visual feature extraction. The threshold determination of the edge gradient amplitude is combined with the sensor noise characteristics to avoid false detection caused by illumination changes. All parameters (such as dynamic threshold, gradient threshold, and coordinate transformation parameters) are set through quantifiable experimental or statistical methods to ensure the repeatability and objectivity of the technical solution.

[0102] Figure 2 A flowchart for verifying sensitive areas in the present invention is provided. Verifying sensitive areas includes verifying the physical spatial consistency of sensitive areas by correlating characteristic mutations of visual data and millimeter-wave radar data. Specifically,

[0103] Count the number of pixels whose edge gradient amplitude exceeds the preset gradient threshold;

[0104] Calculate the spatial distribution matching degree between the velocity mutation amplitude in the radar sensitive area and the number of edge gradient pixels in the visual sensitive area;

[0105] If the spatial distribution matching degree exceeds the preset matching threshold and the overlap between the pixel coordinate area and the visual sensitive area exceeds the preset ratio, it is judged that the physical space is consistent and the sensitive area verification passes; otherwise, it is marked as an invalid area and eliminated, and the sensitive area verification fails.

[0106] When verifying the physical spatial consistency of sensitive areas, verification is performed based on the marked candidate sensitive areas. Candidate sensitive areas include radar sensitive areas and visual sensitive areas. Radar sensitive areas are areas where the velocity change rate extracted by time-domain filtering of millimeter-wave radar data exceeds a dynamic threshold. Visual sensitive areas are areas where the gradient amplitude extracted by edge gradient mutation features of visual data exceeds a preset gradient threshold.

[0107] The target coordinates in the radar coordinate system of the radar sensitive area are mapped to the visual sensor image plane using spatial coordinate transformation parameters, generating the corresponding pixel coordinate area. The spatial coordinate transformation parameters are calculated by the ratio of the spatial compensation component to the visual sensor resolution. For example, when the spatial compensation component is 120 pixels / meter and the visual sensor resolution is 200 pixels / meter, the coordinate transformation parameter is 0.6. The target coordinates in the radar coordinate system are converted to pixel coordinates on the image plane using an affine transformation matrix. The affine transformation matrix is ​​pre-acquired through calibration experiments. For example, a checkerboard calibration plate is used to collect visual images and radar point cloud data while the vehicle is stationary. The translation, rotation, and scaling parameters of the transformation matrix are calculated using the least squares method to ensure that the coordinate mapping error is less than 0.5 pixels.

[0108] Calculate the spatial distribution matching degree between the radar-sensitive area and the visual-sensitive area. The velocity mutation amplitude in the radar-sensitive area is the velocity change rate extracted by time-domain filtering. For example, the velocity change rate in a certain area is 40 m / s². The number of edge gradient pixels in the visual-sensitive area is the total number of pixels whose gradient amplitude exceeds the preset gradient threshold. For example, the number of pixels in a certain area whose gradient amplitude exceeds a threshold of 50 is 200. Spatial distribution matching is calculated using the Pearson correlation coefficient. For example, the correlation coefficient between the velocity mutation amplitude distribution in the radar-sensitive area and the number of gradient pixels in the visual-sensitive area is 0.7. The preset matching threshold is set based on historical data statistics. For example, the mean correlation coefficient of 1000 valid sensitive areas is 0.65, with a standard deviation of 0.05, and the threshold is set to 0.6.

[0109] Calculate the overlap between the pixel coordinates of the radar-sensitive area and the visual-sensitive area. The overlap is determined by the ratio of the intersection area to the radar-sensitive area. For example, the intersection area is 85% of the radar area. The preset ratio is set based on statistical calibration experiments. For example, the mean overlap of valid sensitive areas in 100 sets of calibration data is 82%, with a standard deviation of 3%. The preset ratio is set to 80%. If the spatial distribution match exceeds 0.6 and the overlap exceeds 80%, the physical space is considered consistent; otherwise, the area is marked as invalid and discarded.

[0110] Among them, the radar sensitive area is the area where the speed change rate exceeds the dynamic threshold extracted by time domain filtering, and the dynamic threshold is the top 5% percentile of the speed mutation amplitude of high-speed scenes in historical data; the visual sensitive area is the area where the gradient amplitude detected by the Sobel operator exceeds the preset gradient threshold, and the preset gradient threshold is set according to the dark field noise measurement value of the sensor; the Pearson correlation coefficient is a well-known statistical method used to quantify the linear correlation between two sets of data, and the calculation formula is covariance divided by the product of standard deviations; the intersection area ratio is obtained by calculating the ratio of the number of overlapping pixels in the two areas to the total number of pixels in the radar sensitive area; the calibration experiment statistics include data covering sunny days, rainy days, and night scenes to ensure the universality of the threshold.

[0111] For verified sensitive areas, pseudo-sensitive areas are eliminated based on historical trajectory data, and high-confidence feature segments are screened based on the correlation between visual features and millimeter-wave radar velocity distribution, including:

[0112] Based on the continuity of the target acceleration rate in the historical trajectory data, a physically feasible range of the acceleration rate is constructed;

[0113] Eliminate pseudo-sensitive areas where the acceleration change rate exceeds the physically feasible range. At the same time, perform multi-frame trajectory smoothness verification on sensitive areas where the acceleration change rate is at the boundary of the range. If the number of sudden changes in acceleration direction within several consecutive frames exceeds the preset number, it is determined to be a pseudo area.

[0114] Extract visual feature segments from verified sensitive areas, calculate the spatial covariance between the corresponding edge gradient directional distribution and the millimeter-wave radar velocity directional distribution, and extract the covariance benchmark value for similar scenarios in historical trajectories.

[0115] The fusion weight is dynamically assigned based on the deviation between the current spatial covariance and the covariance baseline value. The smaller the deviation, the higher the visual feature weight, and the larger the deviation, the higher the millimeter wave radar speed weight.

[0116] The spatial and temporal features of vision and radar in the multimodal feature segments are corrected bidirectionally. The visual features correct the spatial positioning jump of the radar velocity distribution, and the radar velocity corrects the motion blur artifacts of the visual features, thus generating high-confidence feature segments that are consistent in time and space.

[0117] When eliminating pseudo-sensitive areas and screening high-confidence feature segments based on historical trajectory data, a physically feasible range for the target acceleration change rate is constructed. The upper limit of the physically feasible range is set in combination with the vehicle's maximum dynamic performance and safety redundancy. For example, when the maximum acceleration of a car is 10 meters per second squared, the upper limit is set to 11 meters per second cubed. The lower limit is set by analyzing historical uniform speed scene data. For example, statistics show that the acceleration change rate in uniform speed scenes is usually less than 0.5 meters per second cubed, so the lower limit is set to 0.3 meters per second cubed. Sensitive areas outside this range are eliminated. For example, if the acceleration change rate in a certain area is significantly higher than the upper limit, it is determined to be a pseudo-area.

[0118] For sensitive areas where the rate of change of acceleration is at the boundary of the interval, multi-frame trajectory smoothness verification is performed. If a sensitive area frequently experiences sudden changes in acceleration direction (for example, a sudden change from due north to due south) in three consecutive frames of data, and the number of such changes exceeds the historically reasonable value (for example, the reasonable number of changes in high-speed scenarios should not exceed one), it is considered a pseudo-area. A sudden change in acceleration direction is defined as a change in direction of more than 90 degrees between adjacent frames. For example, if an area is moving eastward in the first frame and suddenly changes to westward in the second frame, this is considered a sudden change in direction.

[0119] Extract visual feature segments from verified sensitive areas and calculate the spatial correlation between their edge gradient directional distribution and the millimeter-wave radar velocity directional distribution. The edge gradient direction is obtained through visual image processing, such as detecting the directional distribution of lane lines or obstacle edges in the image. The millimeter-wave radar velocity direction is determined by the angle of the target motion vector, for example, the motion direction of the vehicle in front is 20 degrees east of south. Spatial correlation is assessed by statistically analyzing the consistency of the two types of directional distributions. For example, when the visual and radar directional distributions are highly consistent, the correlation is strong, and vice versa. The baseline correlation of historical similar scenes is obtained through classification statistics. For example, the correlation between the visual and radar directions in curved scenes is generally higher than that in straight scenes.

[0120] Fusion weights are dynamically assigned based on the degree of deviation between the current correlation and the historical benchmark. If the current correlation is close to the historical benchmark (for example, the correlation for a curve scene is 0.8 and the current value is 0.75), the visual feature weight is set to 0.8 and the radar weight is set to 0.2. If the deviation is large (for example, the current value is 0.5), the radar weight is set to 0.8. The weight adjustment rule is determined by fitting historical data. For example, for every 10% increase in deviation, the radar weight increases by 0.2.

[0121] Bidirectional corrections are performed on multimodal feature segments. When the radar detects an unusual jump in the position of a target between consecutive frames (e.g., a sudden change from the left lane to the right lane), while the visual data shows the target moving continuously along the lane line, the radar positioning is corrected based on the visual features. When the visual data exhibits blurring artifacts due to high-speed target motion (e.g., blurred vehicle outline edges), while the radar velocity distribution shows uniform target motion, the radar data is used to correct the motion trajectory of the visual features. The corrected feature segments must meet spatiotemporal consistency, for example, the target position must be continuous and smooth, and the direction of motion must conform to physical laws.

[0122] The safety redundancy factor is dynamically adjusted according to the vehicle type, for example, 10% for sedans and 15% for trucks; historical benchmark relevance is stored by scene classification, for example, benchmark values ​​are set separately for curves, straight roads, and congested scenes; the weight adjustment rule is generated by analyzing the relationship between correlation deviation and fusion accuracy in historical data; abnormal jump judgment is that the position jump amplitude exceeds 1.5 times the visual continuous motion interpolation result and is considered abnormal; motion blur correction is triggered when the gradient direction discreteness of the visual blur area exceeds the radar speed direction discreteness.

[0123] The physically feasible interval incorporates vehicle dynamics parameters (such as maximum acceleration) and historical statistical patterns, ensuring that the screening criteria align with actual driving behavior. Multi-frame verification strengthens single-frame detection results, eliminating transient noise interference through temporal continuity constraints. Correlation weight allocation relies on established cross-modal feature association rules to ensure the objectivity of dynamic adjustments. Bidirectional correction leverages the complementary physical properties of different modalities, for example, visual spatial stability corrects for radar hopping, while radar motion accuracy corrects for visual blur. The final output is a high-confidence feature segment that meets spatiotemporal consistency constraints.

[0124] Decouple high-confidence feature segments, suppress visual motion artifacts, and correct spatial positioning jumps in millimeter-wave radar data to generate aligned multimodal feature vectors, including:

[0125] Decompose high-confidence feature segments based on the spatiotemporal dimension to separate the motion artifact components in visual features from the positioning jump components in millimeter-wave radar features;

[0126] Perform gradient direction filtering on motion artifact components to suppress gradient components that deviate from the radar velocity distribution direction by more than a preset angle;

[0127] The positioning jump components are interpolated spatially and continuously, and the interpolation step size is dynamically adjusted according to the consistency of target displacement in adjacent frames in the visual features;

[0128] The filtered visual features and the interpolated radar features are weightedly concatenated according to the spatiotemporal baseline compensation coefficient to generate a channel-aligned multimodal feature vector.

[0129] Based on the channel weight distribution of similar scenes in historical trajectories, the multimodal feature vector is normalized and energy calibrated.

[0130] When decoupling high-confidence feature segments, visual features and millimeter-wave radar features are decomposed based on the spatiotemporal dimensions. This spatiotemporal decomposition utilizes sliding window covariance analysis, with the window length matching the preset time window length. For example, the temporal covariance matrix of the feature segments is calculated within a 0.2-second window. The eigenvalues ​​of the covariance matrix are then extracted to separate visual motion artifacts from radar positioning jump components. For example, high-frequency temporal components in visual features correspond to motion artifacts, while low-frequency spatial components in radar features correspond to positioning jumps. The decomposition logic is as follows: If the temporal covariance of a feature component is significantly higher than its spatial covariance, it is determined to be a visual motion artifact; if the spatial covariance is significantly higher than its temporal covariance, it is determined to be a radar positioning jump.

[0131] Gradient directional filtering is performed on the separated visual motion artifact components to suppress gradient components that deviate from the radar velocity distribution direction by more than a preset angle. The preset angle is set based on the directional consistency statistics of effective sensitive areas in historical data. For example, if statistics show that the directional deviation of 95% of effective areas is less than 15 degrees, the preset angle is set to 15 degrees. Gradient directional filtering is achieved through a directionally selective mask. For example, only gradient components that deviate from the radar velocity direction within ±15 degrees are retained, and the remaining components are set to zero. If the visual gradient direction of a certain area is 20 degrees east of north, while the radar velocity direction is 10 degrees east of south, a deviation of 30 degrees, this gradient component is suppressed.

[0132] Millimeter-wave radar positioning jump components are spatially interpolated continuously, with the interpolation step size dynamically adjusted based on the consistency of target displacement between adjacent frames as determined by visual features. For example, if visual data shows a target displacement of 3 meters between two consecutive frames, while radar data shows a 5-meter displacement jump, the interpolation step size is set to 3 meters based on visual displacement consistency, and cubic spline interpolation is used to correct the radar positioning data to a continuous 3-meter displacement. If the target moves violently (such as during sudden braking), the interpolation algorithm switches to linear interpolation to prevent overfitting.

[0133] The filtered visual features and interpolated radar features are weightedly concatenated using the spatiotemporal baseline compensation coefficients to generate a channel-aligned multimodal feature vector. The weighted concatenation weight is calculated by combining the dynamically assigned fusion weights with the spatiotemporal baseline compensation coefficients. For example, if a visual weight of 0.7 and a compensation coefficient of 0.8 are assigned, the final concatenation weight is 0.7 × 0.8 = 0.56. Channel alignment is achieved by normalizing the spatiotemporal resolution. For example, millimeter-wave radar data is upsampled from 50 Hz to the visual sensor's 30 Hz sampling rate and linear interpolation is used to fill in the intermediate frames.

[0134] Based on the channel weight distribution of similar scenes in historical trajectories, normalized energy calibration is performed on multimodal feature vectors. This historical channel weight distribution is generated through cluster analysis. For example, scenes are categorized as straight roads, curves, and congestion. The average visual and radar weights for each scene type are calculated as the calibration benchmark. The calibration method calculates the ratio of the energy of each channel in the current feature vector to the historical average. If the current visual channel energy is 1.2 times the historical average, the scaling factor is 0.83 (1 / 1.2) to ensure that the calibrated energy is consistent with the historical benchmark. Energy calibration avoids energy imbalances between modalities caused by sudden changes in illumination or speed.

[0135] Covariance significance is determined as a visual artifact when the temporal covariance is greater than twice the spatial covariance, and as a radar jump when the spatial covariance is greater than twice the temporal covariance; the interpolation algorithm switching condition is to enable linear interpolation when the target acceleration change rate exceeds the upper limit of the physically feasible range; the channel alignment interpolation method uses linear interpolation for upsampling and mean pooling for downsampling; historical weight statistics cover sunny, rainy, and nighttime scenes, with a statistical period of three months; the energy scaling factor is calculated by dividing the current energy by the historical mean, and the coefficient is limited to a range of 0.5-2.0 to avoid extreme scaling.

[0136] The trajectory prediction model is constructed based on the dynamic allocation of fusion weights based on multimodal feature vectors, and the spatiotemporal baseline compensation coefficient is reversely corrected by the residual error between the predicted trajectory and the measured trajectory, including:

[0137] Based on the channel weight distribution of historical multimodal feature vectors, the trajectory prediction model is constructed by dynamically allocating fusion weights. The weight allocation rule is generated based on the historical scene clustering results and the current feature deviation.

[0138] Extract the residual error between the predicted trajectory and the measured trajectory. The residual error includes the horizontal displacement deviation, direction angle deviation and velocity change rate difference.

[0139] The spatial and temporal covariance analysis of the residual error is performed to separate the systematic error components caused by the deviation of the spatial and temporal baseline compensation coefficient.

[0140] According to the amplitude direction of the systematic error component, the time compensation component and the space compensation component in the spatiotemporal baseline compensation coefficient are reversely adjusted;

[0141] The corrected spatiotemporal baseline compensation coefficients are iteratively input into the spatiotemporal correlation model to update the multimodal data alignment parameters.

[0142] When constructing a trajectory prediction model by dynamically allocating fusion weights based on multimodal feature vectors, weights must be dynamically adjusted based on historical scene classification and real-time data features. Historical scene classification results inherit the cluster analysis results. For example, driving scenarios can be categorized into curves, straights, congestion, and emergency braking. Each scenario corresponds to the mean and standard deviation of historical channel weights. The deviation of real-time data features is calculated by calculating the difference between the current channel weight and the mean of similar historical scenarios. For example, if the current visual weight is 0.8, the mean of historical curve scenes is 0.7, and the standard deviation is 0.05, then the deviation is (0.8-0.7) / 0.05 = 2.0. The weight allocation rule is as follows: if the absolute value of the deviation is ≤1, the original weight is maintained; if it is 1 < deviation ≤2, the weight is adjusted by ±0.05; and if it is >2, the weight is adjusted by ±0.1. For example, if the deviation is 2.0, the visual weight is adjusted from 0.7 to 0.8, and the radar weight is adjusted from 0.3 to 0.2.

[0143] The residual error between the predicted and measured trajectories must be quantified in multiple dimensions. Horizontal displacement deviation is calculated as the Euclidean distance between the predicted and measured trajectory points. Directional angle deviation is calculated as the angle between the predicted direction vector and the measured vector. For example, if the predicted direction is 30 degrees east of north and the measured direction is 25 degrees east of north, the deviation is 5 degrees. The velocity rate difference is the difference between the predicted and measured accelerations. For example, if the predicted acceleration is 3 meters per second squared and the measured acceleration is 2.5 meters per second squared, the difference is 0.5 meters per second squared. All residual dimensions must strictly correspond to the raw data fields collected by the sensor to ensure the physical interpretability of the error analysis.

[0144] Covariance analysis of the spatiotemporal dimensions of the residual error is used to isolate systematic error components. The window length of the spatiotemporal covariance analysis is consistent with the preset time window. For example, the covariance matrix of horizontal displacement, directional angle, and velocity differences is calculated within a 0.2-second window. Eigenvalue decomposition of the covariance matrix is ​​used to identify the dominant error component. For example, if the horizontal displacement covariance accounts for 70% of the sum of the eigenvalues, the spatial compensation component deviation is determined to be the primary error source. The covariance significance threshold is set based on historical data statistics. For example, after statistically analyzing the covariance distribution of 1,000 valid scenarios, the mean plus twice the standard deviation is taken as the significance threshold. Exceeding this threshold triggers a correction of the compensation coefficient.

[0145] Correction of systematic error components requires a consideration of both the direction and magnitude of the error. For example, if horizontal displacement covariance analysis indicates an underestimation of the spatial compensation component by 0.3 meters per pixel, the spatial compensation coefficient should be increased by 0.3 meters per pixel. If velocity difference covariance analysis indicates an overestimation of the temporal compensation component by 0.1 seconds per second squared, the temporal compensation coefficient should be reduced by 0.1 seconds. The magnitude of the correction is linearly proportional to the error magnitude, with the proportionality factor derived through regression analysis of historical data. For example, for every 0.1 increase in the error magnitude, the compensation coefficient should be adjusted by 0.05. The direction of the correction should be opposite to the direction of the error; for example, a positive deviation corresponds to a decrease in the compensation coefficient, while a negative deviation corresponds to an increase.

[0146] The revised spatiotemporal baseline compensation coefficients are iteratively input into the spatiotemporal correlation model to update the multimodal data alignment parameters. The iteration trigger is determined by the ratio of the mean residual to the sensitive area determination threshold. For example, if the sensitive area threshold is 2 meters, an iteration is triggered when the mean residual reaches 1 meter (50%). Data alignment and sensitive area identification are re-executed using the updated compensation coefficients. For example, after adjusting the spatial compensation coefficient from 1.2 meters per pixel to 1.5 meters, the pixel coordinate mapping error for the same target is reduced from 5 pixels to 2 pixels. Iterations are terminated when the mean residual falls below the threshold by 20% or when the maximum number of iterations (e.g., 10) is reached to ensure algorithm convergence.

[0147] Historical scene classification is achieved through K-means clustering, and the number of categories is set according to the complexity of the driving scene. For example, highway scenes are divided into 4 categories. The deviation calculation rule is normalization to avoid dimensional differences. For example, the visual weight and radar weight are calculated separately and then weighted averaged. The covariance eigenvalue decomposition uses the Jacobi iteration method to extract the principal components to ensure computational efficiency. The correction amplitude proportional coefficient is generated by fitting the relationship between historical errors and compensation coefficients through linear regression analysis. The iteration termination condition is that the maximum number of iterations is calibrated through experiments to avoid infinite loops.

[0148] The above formulas are all dimensionless and numerical calculations. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0149] It should be noted that the present invention can be deployed on the device itself to implement embedded applications, and can also be run on a PC or other terminal with a user interface, thereby meeting various hardware environments and usage requirements.

[0150] The above embodiments can be implemented in whole or in part via software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in the embodiments of this application are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0151] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0153] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0154] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0155] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0156] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0157] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A vehicle monitoring system based on deep learning, characterized in that: include: Spatiotemporal prediction module: collects visual data and millimeter-wave radar data, and predicts spatiotemporal distribution patterns based on motion parameters in the millimeter-wave radar data; Compensation calculation module: dynamically generates spatiotemporal baseline compensation coefficients for visual data and millimeter-wave radar data based on spatiotemporal distribution patterns; Correlation Verification Module: Builds a spatiotemporal correlation model of multimodal data streams based on spatiotemporal baseline compensation coefficients to identify and verify sensitive areas in high-speed scenarios; Feature screening module: For verified sensitive areas, pseudo-sensitive areas are eliminated based on historical trajectory data, and high-confidence feature segments are screened based on the correlation between visual features and millimeter-wave radar velocity distribution; Decoupling correction module: decouples high-confidence feature segments, suppresses visual motion artifacts, corrects spatial positioning jumps in millimeter-wave radar data, and generates aligned multimodal feature vectors; Closed-loop optimization module: The trajectory prediction model is constructed based on the dynamic allocation of fusion weights based on the multimodal feature vectors, and the spatiotemporal baseline compensation coefficient is reversely corrected through the residual error between the predicted trajectory and the measured trajectory.

2. The vehicle monitoring system based on deep learning according to claim 1, characterized in that: Verifying the sensitive area includes verifying the physical spatial consistency of the sensitive area by correlating the characteristic mutations of the visual data with the millimeter-wave radar data.

3. The vehicle monitoring system based on deep learning according to claim 1, characterized in that: Collect visual data and millimeter-wave radar data, and predict spatiotemporal distribution patterns based on motion parameters in the millimeter-wave radar data, including: Vision data and millimeter-wave radar data are collected and pre-processed synchronously through hardware synchronization trigger signals; Decompose the motion parameters in millimeter-wave radar data into acceleration change rate and motion direction offset; The phase offset in the time dimension is calculated based on the acceleration change rate, and the projection deviation in the space dimension is calculated based on the motion direction offset; Perform time-frequency analysis on the phase offset and projection deviation to extract the high-energy fluctuation interval in the frequency domain characteristics; According to the geometric projection relationship between the high-energy fluctuation range and the sensor installation location, a spatiotemporal distribution pattern including continuous error and mutation error classification is generated.

4. The vehicle monitoring system based on deep learning according to claim 1, characterized in that: Dynamically generate spatiotemporal baseline compensation coefficients for visual data and millimeter-wave radar data based on spatiotemporal distribution patterns, including: Calculating a time compensation component based on the phase offset in the time dimension, wherein the time compensation component is the product of the phase offset and the sampling interval; Calculating a spatial compensation component based on the projection deviation of the spatial dimension, where the spatial compensation component is the ratio of the projection deviation to the sensor installation spacing; The time compensation component and the space compensation component are linearly weighted fused to generate the spatiotemporal baseline compensation coefficient. The weight of the linear weighted fusion is dynamically allocated according to the ratio of continuous error to sudden error in the spatiotemporal distribution pattern.

5. The vehicle monitoring system based on deep learning according to claim 1, characterized in that: A spatiotemporal correlation model of multimodal data streams is constructed based on spatiotemporal baseline compensation coefficients to identify sensitive areas in high-speed scenarios, including: Adjust the time interpolation step and spatial coordinate transformation parameters of the multimodal data stream based on the spatiotemporal baseline compensation coefficient; Perform time-domain filtering on the velocity mutation features in millimeter-wave radar data to extract candidate sensitive areas where the velocity change rate exceeds the dynamic threshold; The radar coordinates of the candidate sensitive area are mapped to the visual sensor image plane through the spatial coordinate transformation parameters to generate the corresponding pixel coordinate area; The edge gradient mutation features of the mapped pixel area in the visual data are extracted, and the area where the edge gradient amplitude exceeds the preset gradient threshold is marked as a sensitive area.

6. The vehicle monitoring system based on deep learning according to claim 5, characterized in that: The time interpolation step is generated by the time compensation component and the preset interpolation window length, and the spatial coordinate transformation parameters are generated by associating the spatial compensation component with the visual sensor resolution.

7. The vehicle monitoring system based on deep learning according to claim 2, characterized in that: The physical spatial consistency of sensitive areas is verified by correlating the characteristic mutations of visual data and millimeter-wave radar data. Specifically: Count the number of pixels whose edge gradient amplitude exceeds the preset gradient threshold; Calculate the spatial distribution matching degree between the velocity mutation amplitude in the radar sensitive area and the number of edge gradient pixels in the visual sensitive area; If the spatial distribution matching degree exceeds the preset matching threshold and the overlap between the pixel coordinate area and the visual sensitive area exceeds the preset ratio, it is determined that the physical space is consistent and the sensitive area verification is passed; Otherwise, it is marked as an invalid area and removed, and the sensitive area verification fails.

8. The vehicle monitoring system based on deep learning according to claim 1, characterized in that: For verified sensitive areas, pseudo-sensitive areas are eliminated based on historical trajectory data, and high-confidence feature segments are screened based on the correlation between visual features and millimeter-wave radar velocity distribution, including: Based on the continuity of the target acceleration rate in the historical trajectory data, a physically feasible range of the acceleration rate is constructed; Eliminate pseudo-sensitive areas where the acceleration change rate exceeds the physically feasible range. At the same time, perform multi-frame trajectory smoothness verification on sensitive areas where the acceleration change rate is at the boundary of the range. If the number of sudden changes in acceleration direction within several consecutive frames exceeds the preset number, it is determined to be a pseudo area. Extract visual feature segments from verified sensitive areas, calculate the spatial covariance between the corresponding edge gradient directional distribution and the millimeter-wave radar velocity directional distribution, and extract the covariance benchmark value for similar scenarios in historical trajectories. The fusion weight is dynamically assigned based on the deviation between the current spatial covariance and the covariance baseline value. The smaller the deviation, the higher the visual feature weight, and the larger the deviation, the higher the millimeter wave radar speed weight. The spatial and temporal features of vision and radar in the multimodal feature segments are corrected bidirectionally. The visual features correct the spatial positioning jump of the radar velocity distribution, and the radar velocity corrects the motion blur artifacts of the visual features, thus generating high-confidence feature segments that are consistent in time and space.

9. The vehicle monitoring system based on deep learning according to claim 1, characterized in that: Decouple high-confidence feature segments, suppress visual motion artifacts, and correct spatial positioning jumps in millimeter-wave radar data to generate aligned multimodal feature vectors, including: Decompose high-confidence feature segments based on the spatiotemporal dimension to separate the motion artifact components in visual features from the positioning jump components in millimeter-wave radar features; Perform gradient direction filtering on motion artifact components to suppress gradient components that deviate from the radar velocity distribution direction by more than a preset angle; The positioning jump components are interpolated spatially and continuously, and the interpolation step size is dynamically adjusted according to the consistency of target displacement in adjacent frames in the visual features; The filtered visual features and the interpolated radar features are weightedly concatenated according to the spatiotemporal baseline compensation coefficient to generate a channel-aligned multimodal feature vector. Based on the channel weight distribution of similar scenes in historical trajectories, the multimodal feature vector is normalized and energy calibrated.

10. The vehicle monitoring system based on deep learning according to claim 1, characterized in that: The trajectory prediction model is constructed based on the dynamic allocation of fusion weights based on multimodal feature vectors, and the spatiotemporal baseline compensation coefficient is reversely corrected by the residual error between the predicted trajectory and the measured trajectory, including: Based on the channel weight distribution of historical multimodal feature vectors, the trajectory prediction model is constructed by dynamically allocating fusion weights. The weight allocation rule is generated based on the historical scene clustering results and the current feature deviation. Extract the residual error between the predicted trajectory and the measured trajectory. The residual error includes the horizontal displacement deviation, direction angle deviation and velocity change rate difference. The spatial and temporal covariance analysis of the residual error is performed to separate the systematic error components caused by the deviation of the spatial and temporal baseline compensation coefficient. According to the amplitude direction of the systematic error component, the time compensation component and the space compensation component in the spatiotemporal baseline compensation coefficient are reversely adjusted; The corrected spatiotemporal baseline compensation coefficients are iteratively input into the spatiotemporal correlation model to update the multimodal data alignment parameters.

Images