Vehicle-mounted monitoring system based on deep learning

Abstract

The invention discloses a vehicle-mounted monitoring system based on deep learning, particularly relates to the technical field of vehicle-mounted multi-mode cooperative sensing, and is used for solving the problems of space-time progressive deviation accumulation and high-speed scene target state estimation distortion caused by hardware synchronization limitation in existing vision and millimeter wave radar fusion. According to the method, high-precision alignment and self-adaptive correction of multi-modal data are realized through dynamic space-time baseline compensation and a closed-loop feedback optimization mechanism; a dynamic compensation coefficient is generated based on sensor space-time distribution mode prediction, and a nonlinear correlation model of a multi-modal data stream is constructed to eliminate sampling frequency difference and transmission fluctuation interference; visual motion artifacts and radar positioning jump noise are separated through a cross-modal feature decoupling technology, and high-confidence feature fragments are extracted in combination with historical trajectory screening and real-time residual analysis; and a closed-loop architecture fusing weight distribution and compensation parameter reverse correction is established, and dynamic calibration and error traceability of multi-source data collaborative sensing are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle-mounted multi-modal collaborative perception, and more specifically, the present invention relates to a vehicle-mounted monitoring system based on deep learning. Background Art

[0002] The rapid development of intelligent driving technology has gradually adopted an architecture in which multi-modal sensors work together in the vehicle-mounted environment to improve the target detection and trajectory prediction capabilities in complex scenarios; among them, the fusion scheme of visual sensors and millimeter-wave radars has become the mainstream technical route due to their complementary characteristics. Visual sensors can provide rich texture information, and millimeter-wave radars have accurate speed measurement and anti-interference capabilities. The two working together can effectively cope with situations such as changes in lighting and occlusion; existing technologies usually align and fuse multi-modal data based on the assumption of time synchronization, relying on hardware clock synchronization or fixed delay compensation strategies to achieve the spatio-temporal correlation of sensor data.

[0003] However, in practical applications, due to the differences in the hardware acquisition frequencies of different sensors and the dynamic fluctuations of the data transmission links, it is difficult to keep the timestamps of multi-modal data streams strictly aligned, resulting in progressive deviations in the spatio-temporal feature fusion process. Especially in the scenario of high-speed vehicle maneuvers, the progressive deviations will be non-linearly amplified as the perception distance increases, causing the distortion of the target motion state estimation, and seriously restricting the decision-making reliability and safety of vehicle-mounted environment perception in critical scenarios. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a vehicle-mounted monitoring system based on deep learning to solve the problems proposed in the above background art.

[0005] To achieve the above object, the present invention provides the following technical solutions: A vehicle-mounted monitoring system based on deep learning, comprising: A spatio-temporal prediction module: collecting visual data and millimeter-wave radar data, and predicting the spatio-temporal distribution pattern based on the motion parameters in the millimeter-wave radar data; A compensation calculation module: dynamically generating spatio-temporal baseline compensation coefficients for visual data and millimeter-wave radar data according to the spatio-temporal distribution pattern; An association verification module: constructing a spatio-temporal association model of multi-modal data streams based on the spatio-temporal baseline compensation coefficients, identifying and verifying sensitive areas in high-speed scenarios; A feature screening module: for the verified sensitive areas, eliminating pseudo-sensitive areas according to historical trajectory data, and screening high-confidence feature segments based on the correlation between visual features and the speed distribution of millimeter-wave radars; Decoupling and correction module: Decouple high-confidence feature segments, suppress visual motion artifacts, and correct spatial positioning jumps in millimeter-wave radar data to generate aligned multi-modal feature vectors. Closed-loop optimization module: Dynamically allocate fusion weights based on multi-modal feature vectors to construct a trajectory prediction model, and reversely correct the spatio-temporal baseline compensation coefficient through the residual error between the predicted trajectory and the measured trajectory.

[0006] In a preferred embodiment, the verification sensitive area includes verifying the physical space consistency of the sensitive area by associating the feature mutations of visual data and millimeter-wave radar data.

[0007] In a preferred embodiment, collect visual data and millimeter-wave radar data, and predict the spatio-temporal distribution pattern based on the motion parameters in the millimeter-wave radar data, including: Synchronously collect visual data and millimeter-wave radar data through a hardware synchronization trigger signal and perform preprocessing; Decompose the motion parameters in the millimeter-wave radar data into the acceleration change rate and the motion direction offset; Calculate the phase offset in the time dimension based on the acceleration change rate, and calculate the projection deviation in the space dimension based on the motion direction offset; Perform time-frequency analysis on the phase offset and the projection deviation, and extract the high-energy fluctuation interval in the frequency domain features; Generate a spatio-temporal distribution pattern including continuous error and mutation error classification according to the geometric projection relationship between the high-energy fluctuation interval and the sensor installation position.

[0008] In a preferred embodiment, dynamically generate the spatio-temporal baseline compensation coefficient of visual data and millimeter-wave radar data according to the spatio-temporal distribution pattern, including: Calculate the time compensation component based on the phase offset in the time dimension, where the time compensation component is the product of the phase offset and the sampling interval; Calculate the space compensation component based on the projection deviation in the space dimension, where the space compensation component is the ratio of the projection deviation to the sensor installation spacing; Linearly weighted fusion of the time compensation component and the space compensation component to generate the spatio-temporal baseline compensation coefficient, and the weights of the linear weighted fusion are dynamically allocated according to the ratio of continuous error and mutation error in the spatio-temporal distribution pattern.

[0009] In a preferred embodiment, construct a spatio-temporal association model for multi-modal data streams based on the spatio-temporal baseline compensation coefficient to identify sensitive areas in high-speed scenarios, including: Adjust the time interpolation step and the spatial coordinate transformation parameters of the multi-modal data stream based on the spatio-temporal baseline compensation coefficient; Perform time-domain filtering on the velocity mutation features in millimeter-wave radar data to extract candidate sensitive regions where the rate of change of velocity exceeds the dynamic threshold; Map the radar coordinates of the candidate sensitive regions to the image plane of the vision sensor through the spatial coordinate transformation parameters to generate the corresponding pixel coordinate regions; Extract the edge gradient mutation features of the mapped pixel regions in the vision data, and mark the regions where the edge gradient magnitude exceeds the preset gradient threshold as sensitive regions.

[0010] In a preferred embodiment, the time interpolation step is generated by the time compensation component and the preset interpolation window length, and the spatial coordinate transformation parameters are generated by associating the spatial compensation component with the resolution of the vision sensor.

[0011] In a preferred embodiment, verify the physical space consistency of the sensitive regions by correlating the feature mutations of the vision data and the millimeter-wave radar data. Specifically: Count the number of pixel points where the edge gradient magnitude exceeds the preset gradient threshold; Calculate the spatial distribution matching degree between the velocity mutation magnitude of the radar sensitive region and the number of edge gradient pixel points in the vision sensitive region; If the spatial distribution matching degree exceeds the preset matching threshold and the overlap degree between the pixel coordinate region and the vision sensitive region exceeds the preset ratio, it is determined that the physical space is consistent and the sensitive region verification passes; otherwise, it is marked as an invalid region and excluded, and the sensitive region verification fails.

[0012] In a preferred embodiment, for the sensitive regions that pass the verification, eliminate the false sensitive regions according to the historical trajectory data, and screen the high-confidence feature segments based on the correlation between the vision features and the millimeter-wave radar velocity distribution, including: Based on the continuity of the target acceleration change rate in the historical trajectory data, construct a physically feasible interval for the acceleration change rate; Eliminate the false sensitive regions where the acceleration change rate exceeds the physically feasible interval. At the same time, perform multi-frame trajectory smoothness verification on the sensitive regions where the acceleration change rate is at the interval boundary. If the number of acceleration direction mutations exceeds the preset number within several consecutive frames, it is determined as a false region; Extract the vision feature segments of the sensitive regions that pass the verification, calculate the spatial covariance of the corresponding edge gradient direction distribution and the millimeter-wave radar velocity direction distribution, and extract the covariance reference value in the same type of scenario in the historical trajectory at the same time; Dynamically allocate the fusion weights according to the deviation degree between the current spatial covariance and the covariance reference value. The smaller the deviation degree, the higher the vision feature weight, and the larger the deviation degree, the higher the millimeter-wave radar velocity weight; Bidirectionally correct the spatio-temporal features of vision and radar in the multi-modal feature segments, where the visual features correct the spatial positioning jumps in the radar speed distribution, and the radar speed corrects the motion blur artifacts in the visual features, to generate spatio-temporally consistent high-confidence feature segments.

[0013] In a preferred embodiment, decouple the high-confidence feature segments to suppress visual motion artifacts and correct the spatial positioning jumps in the millimeter-wave radar data, generating aligned multi-modal feature vectors, including: Decompose the high-confidence feature segments based on spatio-temporal dimensions, separating the motion artifact components in the visual features and the positioning jump components in the millimeter-wave radar features; Perform gradient direction filtering on the motion artifact components to suppress the gradient components whose deviation from the radar speed distribution direction exceeds a preset angle; Perform spatial continuity interpolation on the positioning jump components, and the interpolation step size is dynamically adjusted according to the consistency of target displacements in adjacent frames in the visual features; Weightedly splice the filtered visual features and the interpolated radar features according to the spatio-temporal baseline compensation coefficient to generate channel-aligned multi-modal feature vectors; Perform normalized energy calibration on the multi-modal feature vectors based on the channel weight distribution of similar scenarios in the historical trajectory.

[0014] In a preferred embodiment, construct a trajectory prediction model based on dynamically allocating fusion weights for the multi-modal feature vectors, and reversely correct the spatio-temporal baseline compensation coefficient through the residual error between the predicted trajectory and the measured trajectory, including: Based on the channel weight distribution of historical multi-modal feature vectors, dynamically allocate fusion weights to construct a trajectory prediction model, and the weight allocation rule is generated according to the historical scenario clustering results and the current feature deviation degree; Extract the residual error between the predicted trajectory and the measured trajectory, and the residual error includes horizontal displacement deviation, direction angle deviation, and speed change rate difference; Perform spatio-temporal dimensional covariance analysis on the residual error to separate the systematic error components caused by the deviation of the spatio-temporal baseline compensation coefficient; According to the amplitude direction of the systematic error components, reversely adjust the time compensation component and the spatial compensation component in the spatio-temporal baseline compensation coefficient; Iteratively input the corrected spatio-temporal baseline compensation coefficient into the spatio-temporal association model to update the multi-modal data alignment parameters.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention significantly improves the spatio-temporal feature alignment accuracy and dynamic scene adaptability of in-vehicle environment perception systems through a dynamic spatio-temporal baseline compensation mechanism and a closed-loop feedback optimization architecture for multi-modal data streams; an autonomous modeling and compensation coefficient generation method based on the spatio-temporal distribution pattern of sensor data realizes high-precision dynamic spatio-temporal correlation between visual data and millimeter-wave radar data; by constructing a non-linear compensation model for multi-modal data streams, it effectively eliminates the cumulative progressive spatio-temporal deviation caused by differences in sensor sampling frequencies and transmission delays in high-speed maneuvering scenarios, greatly reducing the risk of distorted motion state estimation of distant targets and enhancing the robustness of perception decisions in complex road conditions. 2. Through a cross-modal feature decoupling correction and self-optimizing fusion mechanism, it enhances the collaborative perception quality of multi-source heterogeneous data; a two-way screening strategy based on physical constraints and historical trajectory features accurately separates visual motion artifacts and radar positioning jump interferences, enabling rapid extraction of high-confidence feature segments; a closed-loop compensation parameter iteration system integrating residual feedback can trace sensor errors in real time and autonomously adjust spatio-temporal alignment parameters in dynamic scenarios, ensuring the long-term stability of multi-modal data fusion, enabling the system to maintain accurate expression and efficient collaboration of multi-dimensional spatio-temporal features under extreme lighting, target occlusion, and sudden maneuver conditions, providing low-latency and highly reliable environment perception inputs for intelligent driving control modules, and enhancing safety redundancy and decision-making fault tolerance in complex traffic scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic structural diagram of an in-vehicle monitoring system based on deep learning according to the present invention; Figure 2 It is a flow chart for verifying sensitive areas in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0018] Embodiment: Figure 1 A schematic structural diagram of an in-vehicle monitoring system based on deep learning according to the present invention is given. An in-vehicle monitoring system based on deep learning includes: Spatio-temporal prediction module: Collect visual data and millimeter-wave radar data, and predict the spatio-temporal distribution pattern based on the motion parameters in the millimeter-wave radar data; Compensation calculation module: Dynamically generate spatio-temporal baseline compensation coefficients for visual data and millimeter-wave radar data according to the spatio-temporal distribution pattern; Association verification module: Construct a spatio-temporal association model for multimodal data streams based on spatio-temporal baseline compensation coefficients, identify sensitive areas in high-speed scenarios and verify them; Feature screening module: For the verified sensitive areas, eliminate pseudo-sensitive areas according to historical trajectory data, and screen high-confidence feature segments based on the correlation between visual features and the velocity distribution of millimeter-wave radar; Decoupling and correction module: Decouple the high-confidence feature segments, suppress visual motion artifacts and correct the spatial positioning jumps of millimeter-wave radar data to generate aligned multimodal feature vectors; Closed-loop optimization module: Construct a trajectory prediction model based on multimodal feature vectors to dynamically allocate fusion weights, and inversely correct the spatio-temporal baseline compensation coefficients through the residual error between the predicted trajectory and the measured trajectory.

[0019] Collect visual data and millimeter-wave radar data, and predict the spatio-temporal distribution pattern based on the motion parameters in the millimeter-wave radar data, including: Synchronously collect visual data and millimeter-wave radar data through a hardware synchronization trigger signal and perform preprocessing; Decompose the motion parameters in the millimeter-wave radar data into the acceleration change rate and the motion direction offset; Calculate the phase offset in the time dimension based on the acceleration change rate, and calculate the projection deviation in the spatial dimension based on the motion direction offset; Perform time-frequency analysis on the phase offset and the projection deviation, and extract the high-energy fluctuation interval in the frequency domain features; Generate a spatio-temporal distribution pattern including continuous error and mutation error classification according to the geometric projection relationship between the high-energy fluctuation interval and the sensor installation position.

[0020] The hardware synchronization trigger signal is generated by the vehicle-mounted central controller and sent to the visual sensor and the millimeter-wave radar simultaneously through a physical level signal to ensure that the starting time deviation of the data acquisition of the two is less than 1 millisecond. The visual sensor collects image data at a fixed frequency, such as 30 Hz, and the millimeter-wave radar collects target motion parameters at a higher frequency, such as a pulse repetition frequency of 50 Hz. Median filtering is performed on the visual data to eliminate sudden illumination noise, for example, the neighborhood pixels of the image are sorted using a 3×3 pixel window and the median value is used to replace the central pixel value. Doppler frequency shift compensation is performed on the millimeter-wave radar data, for example, the speed measurement value is corrected based on the linear relationship between the target radial speed and the radar wave frequency to eliminate the speed estimation deviation of the moving target.

[0021] The acceleration change rate is obtained by calculating the second derivative of the velocity data at adjacent time points. For example, the velocity change amount is calculated every 0.02 seconds and then differentiated. The motion direction offset is calculated by the sine difference of the target azimuth angles between the current frame and the previous frame. For example, the azimuth angle difference is mapped to the lateral displacement on a two-dimensional plane. The phase offset in the time dimension is calculated based on the acceleration change rate. Specifically, the acceleration change rate is multiplied by the preset time window length to obtain the cumulative phase deviation on the time axis. The preset time window length is set according to the maximum expected acceleration of the vehicle. For example, it is set to 0.2 seconds for ordinary passenger cars. The projection deviation amount in the space dimension is calculated based on the motion direction offset. Specifically, the direction offset is converted into the pixel displacement in the visual sensor coordinate system, and coordinate transformation is performed in combination with the geometric projection relationship between the millimeter-wave radar and the visual sensor installation positions. The geometric projection relationship is pre-acquired through a calibration experiment. For example, the visual image of the calibration board and the radar point cloud data are collected when the vehicle is stationary, and the affine transformation matrix is calculated by the least squares method.

[0022] Time-frequency analysis is performed on the phase offset and the projection deviation. For example, the short-time Fourier transform is used to convert the time-domain signal into a frequency-domain energy distribution map. The window function length of the short-time Fourier transform is set to 0.1 seconds, the overlap rate is 50%, and the frequency resolution is 10 Hz.

[0023] The high-energy fluctuation intervals with energy values exceeding the preset threshold are extracted from the frequency-domain features. For example, the frequency bands with energy values exceeding the statistical threshold of historical data are marked as high-energy intervals. The preset threshold is dynamically adjusted according to the noise level in the normal driving scenario of the vehicle. For example, the 10% quantile value of the frequency-domain energy distribution is taken as the judgment boundary. The spatio-temporal distribution pattern including continuous errors and mutation errors classification is generated according to the geometric projection relationship between the high-energy fluctuation intervals and the sensor installation positions. Continuous errors correspond to the stable fluctuations of the phase offset in a uniform motion scenario. For example, when the vehicle is moving straight at a constant speed, the phase offset fluctuates within ±5 milliseconds. Mutation errors correspond to the step-like jumps of the projection deviation in an emergency acceleration or steering scenario. For example, when braking suddenly, the projection deviation increases by more than 20 pixels within 0.1 seconds.

[0024] The hardware synchronization trigger signal can be a GPIO level signal, a CAN bus trigger instruction, or a PPS synchronization pulse. For example, when using GPIO trigger, the controller outputs a 5V high-level pulse to the sensor trigger pin. The Doppler frequency shift compensation is based on the formula to correct the target radial velocity, where represents the radial velocity of the target object (i.e., the moving speed of the target relative to the millimeter-wave radar), is the echo frequency offset, is the speed of light, is the radar emission frequency. For example, the frequency shift compensation range of a 77GHz millimeter-wave radar is set to ±500Hz. The preset time window length is shortened to 0.1 seconds when the maximum vehicle acceleration is 10m / s². The energy peak interval in the 0-100Hz frequency band is extracted from the frequency-domain energy distribution map. The calibration error of the affine transformation matrix is less than 0.5 pixels. For example, it is solved by corresponding the corner coordinates of the checkerboard calibration board with the three-dimensional coordinates of the radar one by one.

[0025] Hardware synchronization triggering ensures the time alignment of data, providing input with timing consistency for subsequent parameter decomposition. The decomposition of the acceleration change rate and the direction offset decomposes the motion parameters into error sources in the time and space dimensions. Time-frequency analysis focuses on the high-energy interval in the frequency domain, reducing the influence of noise interference on error classification. The geometric projection relationship maps the radar data to the visual coordinate system, ensuring the physical consistency of spatial dimension calculation. The classification results of continuous and mutation errors provide a prior basis for generating dynamic compensation coefficients. For example, linear interpolation compensation is used for continuous errors, and non-linear filtering compensation is used for mutation errors.

[0026] Dynamically generate the spatio-temporal baseline compensation coefficients for visual data and millimeter-wave radar data according to the spatio-temporal distribution pattern, including: Calculate the time compensation component based on the phase offset in the time dimension, where the time compensation component is the product of the phase offset and the sampling interval; Calculate the space compensation component based on the projection deviation in the space dimension, where the space compensation component is the ratio of the projection deviation to the sensor installation spacing; Linearly weighted fuse the time compensation component and the space compensation component to generate the spatio-temporal baseline compensation coefficient, and the weights of the linear weighted fusion are dynamically allocated according to the ratio of continuous errors and mutation errors in the spatio-temporal distribution pattern.

[0027] The spatio-temporal distribution pattern contains the classification results of continuous errors and mutation errors, where continuous errors are the steady fluctuations of the phase offset in the time dimension, and mutation errors are the step jumps of the projection deviation in the space dimension.

[0028] The time compensation component is calculated as the product of the phase offset and the sampling interval. The phase offset is obtained by multiplying the acceleration change rate by the preset time window length. For example, when the acceleration change rate is 2m / s² and the time window length is 0.2 seconds, the phase offset is 0.4 seconds. The sampling interval is the difference between the acquisition periods of the visual sensor and the millimeter-wave radar. For example, when the acquisition period of the visual sensor is 33 milliseconds and the acquisition period of the millimeter-wave radar is 20 milliseconds, the sampling interval is 13 milliseconds. The time compensation component is used to quantify the cumulative effect of the timestamp alignment deviation. For example, the product of the phase offset of 0.4 seconds and the sampling interval of 13 milliseconds is 5.2 milliseconds².

[0029] The calculation method of the spatial compensation component is the ratio of the projection deviation amount to the sensor installation spacing. The projection deviation amount is obtained by mapping the motion direction offset amount to the pixel displacement amount in the visual sensor coordinate system. For example, when the direction offset amount is 3 degrees, the projection deviation amount is 60 pixels.

[0030] The sensor installation spacing is the physical distance between the visual sensor and the millimeter-wave radar, which is obtained through calibration experiments. For example, it is measured using a laser rangefinder and recorded as 0.5 meters. The spatial compensation component is used to quantify the scaling ratio of the spatial coordinate deviation. For example, the ratio of the projection deviation amount of 60 pixels to the installation spacing of 0.5 meters is 120 pixels / meter.

[0031] The linear weighted fusion weights of the time compensation component and the spatial compensation component are dynamically allocated according to the ratio of continuous errors to abrupt errors. The error ratio is calculated by statistically counting the proportion of the number of the two types of errors in the spatio-temporal distribution pattern. For example, when the number of continuous errors is 80 and the number of abrupt errors is 20, the proportion of continuous errors is 80%. When the proportion of continuous errors exceeds 70%, the weight of the time compensation component is set to 0.8 and the weight of the spatial compensation component is set to 0.2; when the proportion of abrupt errors exceeds 60%, the weight of the time compensation component is set to 0.3 and the weight of the spatial compensation component is set to 0.7. The weight allocation strategy is set based on the statistical laws of historical driving scenario data. For example, the proportion of continuous errors is usually higher than 70% in highway scenarios, and the proportion of abrupt errors is usually higher than 60% in urban congestion scenarios.

[0032] The spatio-temporal baseline compensation coefficient is generated through a weighted fusion formula. For example, when the time compensation component is 5.2 ms², the spatial compensation component is 120 pixels / meter, and the weights are 0.8 and 0.2 respectively, the spatio-temporal baseline compensation coefficient is 5.2×0.8 + 120×0.2 = 28.16. The spatio-temporal baseline compensation coefficient is input into the spatio-temporal correlation model to adjust the time interpolation step and the spatial coordinate transformation parameters of the model. For example, the time interpolation step is adjusted from the default 10 ms to the interpolation interval corresponding to the dynamic value of 28.16 ms², and the spatial coordinate transformation parameters are adjusted according to 120 pixels / meter to change the pixel mapping ratio of the target in the visual image.

[0033] Among them, before calculating the spatio-temporal baseline compensation coefficient, it is necessary to perform dimensionless processing on the time compensation component and the spatial compensation component.

[0034] Based on the spatio-temporal baseline compensation coefficient, a spatio-temporal correlation model of multi-modal data streams is constructed to identify sensitive areas in high-speed scenarios, including: Based on the spatio-temporal baseline compensation coefficient, adjust the time interpolation step and the spatial coordinate transformation parameters of the multi-modal data stream. The time interpolation step is generated by the time compensation component and the preset interpolation window length, and the spatial coordinate transformation parameters are generated by correlating the spatial compensation component with the visual sensor resolution; Perform time-domain filtering on the velocity mutation features in millimeter-wave radar data to extract candidate sensitive regions where the rate of velocity change exceeds the dynamic threshold; Map the radar coordinates of the candidate sensitive regions to the image plane of the vision sensor through the spatial coordinate transformation parameters to generate the corresponding pixel coordinate regions; Extract the edge gradient mutation features of the mapped pixel regions in the vision data, and mark the regions where the edge gradient magnitude exceeds the preset gradient threshold as sensitive regions.

[0035] The time compensation component in the spatio-temporal baseline compensation coefficient is calculated by the product of the phase offset and the sampling interval. For example, when the phase offset is 0.4 seconds and the sampling interval is 13 milliseconds, the time compensation component is 5.2 milliseconds². The spatial compensation component is calculated by the ratio of the projection deviation to the sensor installation spacing. For example, when the projection deviation is 60 pixels and the installation spacing is 0.5 meters, the spatial compensation component is 120 pixels / meter.

[0036] The time interpolation step is generated by the product of the time compensation component and the preset interpolation window length. The preset interpolation window length is the same as the preset time window length used to calculate the phase offset. For example, when the preset time window length is 0.2 seconds, the time interpolation step is 5.2 milliseconds²×0.2 seconds = 1.04 seconds. The spatial coordinate transformation parameters are generated by the ratio of the spatial compensation component to the resolution of the vision sensor. For example, when the resolution of the vision sensor is 200 pixels / meter, the spatial coordinate transformation parameters are 120 pixels / meter÷200 pixels / meter = 0.6.

[0037] Perform time-domain filtering on the velocity mutation features in millimeter-wave radar data to extract candidate sensitive regions where the rate of velocity change exceeds the dynamic threshold. The dynamic threshold is set according to the statistical distribution of the velocity mutation amplitudes in high-speed scenarios in historical data. Specifically, the velocity mutation data of the vehicle in the highway scenario in the past month is statistically analyzed, and the 5% quantile value is taken as the dynamic threshold.

[0038] The time-domain filtering uses the sliding window averaging method, and the window length is the same as the preset time window length. For example, the moving average of the rate of velocity change is calculated with a window length of 0.2 seconds, and the regions exceeding the dynamic threshold are marked as candidate sensitive regions. For example, when the rate of velocity change in a certain region increases from 20 m / s to 30 m / s within 0.2 seconds, its rate of velocity change is 50 m / s², exceeding the dynamic threshold of 30 m / s², and it is marked as a candidate sensitive region.

[0039] The target coordinates in the millimeter-wave radar coordinate system of the candidate sensitive region are mapped to the image plane of the vision sensor through the spatial coordinate transformation parameters to generate the corresponding pixel coordinate region. The spatial coordinate transformation parameters are calculated by the ratio of the spatial compensation component to the resolution of the vision sensor. For example, when the spatial compensation component is 120 pixels / meter and the resolution of the vision sensor is 200 pixels / meter, the coordinate transformation parameter is 0.6.

[0040] The target coordinates in the millimeter-wave radar coordinate system are converted into pixel coordinates on the image plane through an affine transformation matrix. The affine transformation matrix is obtained in advance through a calibration experiment. For example, in the vehicle stationary state, the vision image of the checkerboard calibration board and the radar point cloud data are collected, and the six parameters of the transformation matrix are calculated by the least squares method to ensure that the coordinate mapping error is less than 0.5 pixels. For example, the target coordinates (10 meters, 2 meters) in the radar coordinate system are converted into image coordinates (10×0.6, 2×0.6) = (6 pixels, 1.2 pixels).

[0041] Extract the edge gradient mutation features of the mapped pixel region in the vision data, and mark the region where the edge gradient amplitude exceeds the preset gradient threshold as the sensitive region. The edge gradient mutation features are detected by the Sobel operator. The Sobel operator uses a 3×3 convolution kernel to calculate the horizontal and vertical gradient amplitudes of the image region, and the merged result is the edge gradient map. The preset gradient threshold is set according to the noise characteristics of the vision sensor. For example, it is set to 50 under low light conditions and 30 under normal light conditions. The noise characteristics are obtained through a dark field imaging experiment. For example, in a night scene, the edge gradient amplitude of a certain region is 60, which exceeds the preset threshold of 50, so it is marked as a candidate vision sensitive region. If the gradient amplitude of a certain region is 25 in a day scene, which is lower than the threshold of 30, it is not marked as a sensitive region.

[0042] The preset gradient threshold is set based on the measurement result of the noise level of the vision sensor in the dark field experiment. For example, 100 frames of images are collected in a completely dark environment, and the maximum value of the noise gradient amplitude is statistically used as the reference benchmark. The calibration error of the affine transformation matrix is calculated by the mapping residual of the checkerboard corner coordinates, and the residual mean less than 0.5 pixels is regarded as an effective calibration.

[0043] The adjustment of the time interpolation step directly affects the time alignment accuracy of the multi-modal data stream. For example, a longer interpolation step may lead to the accumulation of timestamp deviations. The generation of the spatial coordinate transformation parameters depends on the spatial compensation component and the resolution of the vision sensor to ensure the physical consistency of the radar data to the vision coordinates. The candidate sensitive regions extracted by time-domain filtering are converted into pixel regions on the vision plane through coordinate mapping, providing a spatial positioning basis for subsequent vision feature extraction. The threshold determination of the edge gradient magnitude combines the characteristics of the sensor noise to avoid false detections caused by changes in illumination. All parameters (such as dynamic thresholds, gradient thresholds, coordinate transformation parameters) are set through quantifiable experiments or statistical methods to ensure the repeatability and objectivity of the technical solution.

[0044] Figure 2 The flowchart for verifying the sensitive region in the present invention is given. Verifying the sensitive region includes verifying the physical space consistency of the sensitive region through the feature mutation of associating vision data with millimeter-wave radar data. Specifically: Statistically count the number of pixel points whose edge gradient magnitude exceeds the preset gradient threshold. Calculate the spatial distribution matching degree between the velocity mutation magnitude of the radar sensitive region and the number of edge gradient pixel points in the vision sensitive region. If the spatial distribution matching degree exceeds the preset matching threshold and the overlap degree of the pixel coordinate region and the vision sensitive region exceeds the preset ratio, it is determined that the physical space is consistent and the sensitive region verification passes; otherwise, it is marked as an invalid region and excluded, and the sensitive region verification fails.

[0045] When verifying the physical space consistency of the sensitive region, the verification operation is performed based on the marked candidate sensitive regions. The candidate sensitive regions include the radar sensitive region and the vision sensitive region. The radar sensitive region is the region where the rate of change of velocity extracted by time-domain filtering of millimeter-wave radar data exceeds the dynamic threshold, and the vision sensitive region is the region where the gradient magnitude extracted by the edge gradient mutation feature of vision data exceeds the preset gradient threshold.

[0046] The target coordinates in the radar coordinate system of the radar sensitive region are mapped to the vision sensor image plane through the spatial coordinate transformation parameters to generate the corresponding pixel coordinate region. The spatial coordinate transformation parameters are calculated by the ratio of the spatial compensation component to the resolution of the vision sensor. For example, when the spatial compensation component is 120 pixels / meter and the resolution of the vision sensor is 200 pixels / meter, the coordinate transformation parameter is 0.6. The target coordinates in the radar coordinate system are converted into pixel coordinates on the image plane through an affine transformation matrix, and the affine transformation matrix is obtained in advance through a calibration experiment. For example, when the vehicle is stationary, a checkerboard calibration board is used to collect vision images and radar point cloud data, and the translation, rotation, and scaling parameters of the transformation matrix are calculated by the least squares method to ensure that the coordinate mapping error is less than 0.5 pixels.

[0047] Calculate the spatial distribution matching degree between the radar sensitive area and the visual sensitive area. The velocity mutation amplitude of the radar sensitive area is the velocity change rate extracted by time-domain filtering. For example, the velocity change rate of a certain area is 40 m / s². The number of edge gradient pixels in the visual sensitive area is the total number of pixels whose gradient amplitude exceeds the preset gradient threshold. For example, the number of pixels with a gradient amplitude exceeding the threshold of 50 in a certain area is 200. The spatial distribution matching degree is calculated by the Pearson correlation coefficient. For example, the correlation coefficient between the distribution of the velocity mutation amplitude in the radar sensitive area and the distribution of the number of gradient pixels in the visual sensitive area is 0.7. The preset matching threshold is set according to historical data statistics. For example, the mean value of the correlation coefficients of 1000 groups of effective sensitive areas is 0.65, and the standard deviation is 0.05. The set threshold is 0.6.

[0048] Calculate the overlap degree between the pixel coordinate area of the radar sensitive area and the visual sensitive area. The overlap degree is determined by the ratio of the intersection area to the area of the radar sensitive area. For example, the intersection area is 85% of the radar area. The preset ratio is set by calibration experiment statistics. For example, in 100 groups of calibration data, the mean value of the overlap degrees of effective sensitive areas is 82%, and the standard deviation is 3%. The preset ratio is set to 80%. If the spatial distribution matching degree exceeds 0.6 and the overlap degree exceeds 80%, it is determined that the physical space is consistent; otherwise, it is marked as an invalid area and excluded.

[0049] Among them, the radar sensitive area is the area where the velocity change rate extracted by time-domain filtering exceeds the dynamic threshold, and the dynamic threshold takes the 5% quantile value of the velocity mutation amplitude in the high-speed scene of historical data; the visual sensitive area is the area where the gradient amplitude detected by the Sobel operator exceeds the preset gradient threshold, and the preset gradient threshold is set according to the sensor dark field noise measurement value; the Pearson correlation coefficient is a well-known statistical method used to quantify the linear correlation of two sets of data, and the calculation formula is the covariance divided by the product of the standard deviations; the ratio of the intersection area is obtained by calculating the ratio of the number of overlapping pixels of the two areas to the total number of pixels in the radar sensitive area; the calibration experiment statistics include data covering various scenarios such as sunny days, rainy days, and nights to ensure the universality of the threshold.

[0050] For the sensitive areas that pass the verification, eliminate the false sensitive areas according to the historical trajectory data, and screen out the high-confidence feature segments based on the correlation between the visual features and the millimeter-wave radar velocity distribution, including: Based on the continuity of the target acceleration change rate in the historical trajectory data, construct a physically feasible interval for the acceleration change rate; Eliminate the false sensitive areas where the acceleration change rate exceeds the physically feasible interval. At the same time, perform multi-frame trajectory smoothness verification on the sensitive areas where the acceleration change rate is at the interval boundary. If the number of acceleration direction mutations exceeds the preset number within several consecutive frames, it is determined as a false area; Extract the visual feature segments of the verified sensitive regions, calculate the spatial covariance between the corresponding edge gradient direction distribution and the millimeter-wave radar velocity direction distribution, and extract the covariance reference value in the same type of scenarios in the historical trajectory at the same time; Dynamically allocate the fusion weights according to the deviation degree between the current spatial covariance and the covariance reference value. The smaller the deviation degree, the higher the weight of the visual feature, and the larger the deviation degree, the higher the weight of the millimeter-wave radar velocity; Bidirectionally correct the spatio-temporal features of vision and radar in the multi-modal feature segments. The visual features correct the spatial positioning jumps of the radar velocity distribution, and the radar velocity corrects the motion blur artifacts of the visual features to generate spatio-temporally consistent high-confidence feature segments.

[0051] When eliminating the pseudo-sensitive regions and screening the high-confidence feature segments based on the historical trajectory data, construct a physically feasible interval for the target acceleration change rate. The upper limit of the physically feasible interval is set in combination with the maximum dynamic performance and safety redundancy of the vehicle. For example, when the maximum acceleration of a certain car is 10 m / s², the upper limit is set to 11 m / s². The lower limit is set by analyzing the historical constant-speed scenario data. For example, statistics show that the acceleration change rate in the constant-speed scenario is usually lower than 0.5 m / s², and the lower limit is set to 0.3 m / s². Eliminate the sensitive regions that exceed this interval. For example, if the acceleration change rate of a certain region is significantly higher than the upper limit, it is determined as a pseudo-region.

[0052] For the sensitive regions where the acceleration change rate is at the interval boundary, perform multi-frame trajectory smoothness verification. If the acceleration direction of a certain sensitive region suddenly changes frequently (such as suddenly turning from due north to due south) in three consecutive frames of data, and the number of mutations exceeds the historical reasonable value (such as the reasonable number of mutations in the high-speed scenario does not exceed 1 time), it is determined as a pseudo-region. The definition of the acceleration direction mutation is that the direction change between adjacent frames exceeds 90 degrees. For example, if a certain region moves eastward in the first frame and suddenly changes to westward in the second frame, it is regarded as one direction mutation.

[0053] Extract the visual feature segments of the verified sensitive regions, and calculate the spatial correlation between their edge gradient direction distribution and the millimeter-wave radar velocity direction distribution. The edge gradient direction is obtained through visual image processing. For example, detect the direction distribution of the lane lines or obstacle edges in the image. The millimeter-wave radar velocity direction is determined by the angle of the target motion vector. For example, the motion direction of the vehicle ahead is 20 degrees south by east. The spatial correlation is evaluated by statistically analyzing the consistency degree of the two types of direction distributions. For example, when the visual and radar direction distributions are highly consistent, the correlation is strong, otherwise it is weak. The reference correlation of the historical same type of scenarios is obtained through classification statistics. For example, the visual and radar direction correlations in the curve scenario are usually higher than those in the straight road scenario.

[0054] Dynamically allocate fusion weights according to the deviation of the current relevance from the historical benchmark. If the current relevance is close to the historical benchmark (for example, the relevance of the curve scene is 0.8 and the current value is 0.75), the visual feature weight is set to 0.8 and the radar weight is set to 0.2; if the deviation is large (for example, the current value is 0.5), the radar weight is set to 0.8. The weight adjustment rule is determined by fitting historical data. For example, for every 10% increase in the deviation, the radar weight increases by 0.2.

[0055] Perform two-way correction on multi-modal feature segments. When the radar detects an abnormal jump in the position of a certain target between adjacent frames (for example, suddenly changing from the left lane to the right lane), while the visual data shows that the target continuously moves along the lane line, the radar positioning is corrected based on the visual features. When the visual data shows blurred artifacts due to the high-speed movement of the target (for example, the edges of the vehicle contour are blurred), while the radar speed distribution shows that the target moves at a constant speed, the motion trajectory of the visual features is corrected based on the radar data. The corrected feature segments need to satisfy spatio-temporal consistency. For example, the target position is continuous and smooth, and the motion direction conforms to physical laws.

[0056] The safety redundancy coefficient is dynamically adjusted according to the vehicle type. For example, it is 10% for sedans and 15% for trucks; the historical benchmark relevance is stored by scene classification. For example, the benchmark values are set for curve, straight, and congestion scenes respectively; the weight adjustment rule is generated by fitting the relationship between the relevance deviation and the fusion accuracy in historical data; the abnormal jump determination is that a position jump amplitude exceeding 1.5 times the visual continuous motion interpolation result is regarded as abnormal; the motion blur correction is triggered when the gradient direction dispersion degree of the visual blurred area exceeds the radar speed direction dispersion degree.

[0057] The physically feasible interval inherits vehicle dynamics parameters (such as maximum acceleration) and historical statistical laws to ensure that the screening conditions are consistent with actual driving behaviors. Multi-frame verification strengthens the single-frame detection result and eliminates instantaneous noise interference through time continuity constraints. The correlation weight allocation depends on the established cross-modal feature association rules to ensure the objectivity of dynamic adjustment. The two-way correction is based on the complementarity of the physical characteristics of different modalities. For example, the visual spatial stability corrects the radar jump, and the radar motion accuracy corrects the visual blur, and finally outputs high-confidence feature segments that meet the spatio-temporal consistency constraints.

[0058] Decouple the high-confidence feature segments, suppress visual motion artifacts, and correct the spatial positioning jump of millimeter-wave radar data to generate aligned multi-modal feature vectors, including: Decompose the high-confidence feature segments based on spatio-temporal dimensions, and separate the motion artifact components in the visual features and the positioning jump components in the millimeter-wave radar features; Perform gradient direction filtering on the motion artifact components to suppress the gradient components whose deviation from the radar speed distribution direction exceeds the preset angle; The positioning jump components are interpolated spatially and continuously, and the interpolation step size is dynamically adjusted according to the consistency of target displacement in adjacent frames in the visual features; The filtered visual features and the interpolated radar features are weightedly concatenated according to the spatiotemporal baseline compensation coefficient to generate a channel-aligned multimodal feature vector. Based on the channel weight distribution of similar scenes in historical trajectories, the multimodal feature vector is normalized and energy calibrated.

[0059] When decoupling high-confidence feature segments, visual features and millimeter-wave radar features are decomposed based on the spatiotemporal dimensions. This spatiotemporal decomposition utilizes sliding window covariance analysis, with the window length matching the preset time window length. For example, the temporal covariance matrix of the feature segments is calculated within a 0.2-second window. The eigenvalues of the covariance matrix are then extracted to separate visual motion artifacts from radar positioning jump components. For example, high-frequency temporal components in visual features correspond to motion artifacts, while low-frequency spatial components in radar features correspond to positioning jumps. The decomposition logic is as follows: If the temporal covariance of a feature component is significantly higher than its spatial covariance, it is determined to be a visual motion artifact; if the spatial covariance is significantly higher than its temporal covariance, it is determined to be a radar positioning jump.

[0060] Gradient directional filtering is performed on the separated visual motion artifact components to suppress gradient components that deviate from the radar velocity distribution direction by more than a preset angle. The preset angle is set based on the directional consistency statistics of effective sensitive areas in historical data. For example, if statistics show that the directional deviation of 95% of effective areas is less than 15 degrees, the preset angle is set to 15 degrees. Gradient directional filtering is achieved through a directionally selective mask. For example, only gradient components that deviate from the radar velocity direction within ±15 degrees are retained, and the remaining components are set to zero. If the visual gradient direction of a certain area is 20 degrees east of north, while the radar velocity direction is 10 degrees east of south, a deviation of 30 degrees, this gradient component is suppressed.

[0061] Millimeter-wave radar positioning jump components are spatially interpolated continuously, with the interpolation step size dynamically adjusted based on the consistency of target displacement between adjacent frames as determined by visual features. For example, if visual data shows a target displacement of 3 meters between two consecutive frames, while radar data shows a 5-meter displacement jump, the interpolation step size is set to 3 meters based on visual displacement consistency, and cubic spline interpolation is used to correct the radar positioning data to a continuous 3-meter displacement. If the target moves violently (such as during sudden braking), the interpolation algorithm switches to linear interpolation to prevent overfitting.

[0062] The filtered visual features and the interpolated radar features are weighted and stitched according to the spatio-temporal baseline compensation coefficient to generate a multi-modal feature vector with aligned channels. The weighted stitching weight is jointly calculated by the dynamically allocated fusion weight and the spatio-temporal baseline compensation coefficient. For example, if the visual weight is assigned 0.7 and the compensation coefficient is 0.8, the final stitching weight is 0.7×0.8 = 0.56. Channel alignment is achieved through spatio-temporal resolution normalization. For example, the millimeter-wave radar data is upsampled from a sampling rate of 50Hz to 30Hz of the visual sensor, and linear interpolation is used to fill in the intermediate frame data.

[0063] Based on the channel weight distribution of similar scenarios in the historical trajectory, the multi-modal feature vector is normalized for energy calibration. The historical channel weight distribution is generated through cluster analysis. For example, the scenarios are classified into straight roads, curves, and congestions, and the mean values of the visual and radar weights for each type of scenario are statistically calculated as the calibration benchmark. The calibration method is as follows: calculate the ratio of the energy of each channel of the current feature vector to the historical mean. If the current visual channel energy is 1.2 times the historical mean, the scaling factor is 0.83 (1 / 1.2) to make the calibrated energy consistent with the historical benchmark. Energy calibration avoids the energy imbalance between modalities caused by sudden changes in illumination or speed jumps.

[0064] Covariance significance determination: When the time covariance > 2 times the space covariance, it is determined as a visual artifact; when the space covariance > 2 times the time covariance, it is determined as a radar jump; the switching condition of the interpolation algorithm is to enable linear interpolation when the acceleration change rate of the target exceeds the upper limit of the physically feasible interval; the channel alignment interpolation method is to use linear interpolation for upsampling and mean pooling for downsampling; historical weight statistics cover sunny, rainy, and night scenes, and the statistical period is three months; the energy scaling factor is calculated by the current energy / historical mean, and the coefficient range is restricted to 0.5 - 2.0 to avoid extreme scaling.

[0065] Based on the multi-modal feature vector, a trajectory prediction model is constructed by dynamically allocating fusion weights, and the spatio-temporal baseline compensation coefficient is inversely corrected through the residual error between the predicted trajectory and the measured trajectory, including: Based on the channel weight distribution of the historical multi-modal feature vector, a trajectory prediction model is constructed by dynamically allocating fusion weights, and the weight allocation rule is generated according to the historical scenario clustering result and the current feature deviation; Extract the residual error between the predicted trajectory and the measured trajectory, and the residual error includes horizontal displacement deviation, direction angle deviation, and speed change rate difference; Perform spatio-temporal dimensional covariance analysis on the residual error to separate the systematic error component caused by the deviation of the spatio-temporal baseline compensation coefficient; According to the amplitude direction of the systematic error component, inversely adjust the time compensation component and the space compensation component in the spatio-temporal baseline compensation coefficient; Iteratively input the corrected spatio-temporal baseline compensation coefficients into the spatio-temporal correlation model to update the multi-modal data alignment parameters.

[0066] When constructing a trajectory prediction model by dynamically allocating fusion weights based on multi-modal feature vectors, it is necessary to dynamically adjust the weights in combination with historical scene classification and real-time data features. The historical scene classification results inherit the clustering analysis results. For example, driving scenes are classified into categories such as curves, straight roads, congestion, and emergency braking. Each category of scene corresponds to the mean and standard deviation of the historical channel weights. The deviation of real-time data features is obtained by calculating the difference between the current channel weight and the mean of the same type of historical scene. For example, if the current visual weight is 0.8, the mean of the historical curve scene is 0.7, and the standard deviation is 0.05, then the deviation is (0.8 - 0.7) / 0.05 = 2.0. The weight allocation rule is set as follows: when the absolute value of the deviation ≤ 1, the original weight is maintained; when 1 < deviation ≤ 2, the weight is adjusted by ±0.05; when the deviation > 2, it is adjusted by ±0.1. For example, when the deviation is 2.0, the visual weight is adjusted from 0.7 to 0.8, and the radar weight is adjusted from 0.3 to 0.2.

[0067] The residual error between the predicted trajectory and the measured trajectory needs to be quantified from multiple dimensions. The horizontal displacement deviation is calculated by the Euclidean distance between the predicted trajectory points and the measured trajectory points. The direction angle deviation is calculated by the included angle between the predicted direction vector and the measured vector. For example, if the predicted direction is 30 degrees east of north and the measured is 25 degrees east of north, the deviation is 5 degrees. The difference in the rate of change of speed is the difference between the predicted acceleration and the measured acceleration. For example, if the predicted acceleration is 3 m / s² and the measured is 2.5 m / s², the difference is 0.5 m / s². All residual dimensions need to strictly correspond to the original data fields collected by the sensors to ensure the physical interpretability of the error analysis.

[0068] The spatio-temporal dimensional covariance analysis of the residual error is used to separate the systematic error components. The window length of the spatio-temporal covariance analysis is consistent with the preset time window. For example, the covariance matrix of the horizontal displacement, direction angle, and speed difference is calculated within a 0.2-second window. The eigenvalue decomposition of the covariance matrix is used to identify the dominant error components. For example, if the horizontal displacement covariance accounts for 70% of the total eigenvalue, it is determined that the deviation of the spatial compensation component is the main error source. The covariance significance threshold is set through historical data statistics. For example, after statistically analyzing the covariance distribution of 1000 groups of effective scenes, the mean plus twice the standard deviation is taken as the significance threshold. If it exceeds this threshold, the compensation coefficient is corrected.

[0069] The reverse correction of systematic error components needs to combine the error direction and amplitude. For example, horizontal displacement covariance analysis shows that the spatial compensation component is underestimated by 0.3 meters per pixel. When correcting, the spatial compensation coefficient is increased by 0.3 meters per pixel. The velocity difference covariance shows that the time compensation component is overestimated by 0.1 second per second squared, so the time compensation coefficient is reduced by 0.1 second. The correction amplitude is linearly proportional to the error amplitude, and the proportionality coefficient is obtained through regression analysis of historical data. For example, for every 0.1 increase in the error amplitude, the compensation coefficient is adjusted by 0.05. The correction direction needs to be opposite to the error direction. For example, a positive deviation corresponds to a decrease in the compensation coefficient, and a negative deviation corresponds to an increase.

[0070] The corrected spatio-temporal baseline compensation coefficient needs to be iteratively input into the spatio-temporal correlation model to update the multi-modal data alignment parameters. The iteration trigger condition is set according to the ratio of the residual mean to the sensitivity region determination threshold. For example, when the sensitivity region threshold is 2 meters and the residual mean reaches 1 meter (50%), the iteration is triggered. After the updated compensation coefficient, data alignment and sensitivity region identification are re-executed. For example, after the spatial compensation coefficient is corrected from 1.2 meters per pixel to 1.5 meters, the pixel coordinate mapping error of the same target is reduced from 5 pixels to 2 pixels. The iteration termination condition is that the residual mean is lower than 20% of the threshold or the maximum number of iterations is reached (for example, 10 times) to ensure algorithm convergence.

[0071] Historical scene classification is achieved through K-means clustering. The number of categories is set according to the complexity of the driving scene. For example, highway scenes are divided into 4 categories. The deviation calculation rule is normalization to avoid dimensional differences. For example, the visual weight and radar weight are calculated separately and then weighted and averaged. The covariance eigenvalue decomposition uses the Jacobi iteration method to extract the principal components to ensure calculation efficiency. The correction amplitude proportionality coefficient is generated by fitting the relationship between historical errors and compensation coefficients through linear regression analysis. The iteration termination condition is that the maximum number of iterations is calibrated through experiments to avoid infinite loops.

[0072] The above formulas are all dimensionless and take their numerical values for calculation. The formulas are obtained by collecting a large amount of data for software simulation to get a formula closest to the real situation. The preset parameters and threshold selection in the formulas are set by those skilled in the art according to the actual situation.

[0073] It should be noted that the present invention can be deployed on the device itself to achieve embedded applications, or can also run on a PC with a user interface or other terminals, so as to meet a variety of hardware environments and usage requirements.

[0074] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that contains one or more collections of available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0075] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and modules described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0076] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or modules can be in electrical, mechanical, or other forms.

[0077] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules. They can be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0078] In addition, in each embodiment of the present application, each functional module can be integrated into one processing module, can exist physically alone for each module, or two or more modules can be integrated into one module.

[0079] If the above-mentioned function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0080] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all of them should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0081] Finally: The above is only the preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A vehicle-mounted monitoring system based on deep learning, characterized in that, Including: Space-time prediction module: Collect visual data and millimeter-wave radar data, and predict the space-time distribution pattern based on the motion parameters in the millimeter-wave radar data; Compensation calculation module: Dynamically generate the space-time baseline compensation coefficients for visual data and millimeter-wave radar data according to the space-time distribution pattern; Correlation verification module: Construct a space-time correlation model for multi-modal data streams based on the space-time baseline compensation coefficients, identify sensitive areas in high-speed scenarios and verify them; Feature screening module: For the verified sensitive areas, eliminate false sensitive areas according to historical trajectory data, and screen high-confidence feature segments based on the correlation between visual features and the speed distribution of millimeter-wave radar; Decoupling and correction module; Decouple the high-confidence feature segments, suppress visual motion artifacts and correct the spatial positioning jumps of millimeter-wave radar data to generate aligned multi-modal feature vectors; Closed-loop optimization module: Dynamically allocate fusion weights based on multi-modal feature vectors to construct a trajectory prediction model, and inversely correct the space-time baseline compensation coefficients through the residual error between the predicted trajectory and the measured trajectory.

2. The vehicle-mounted monitoring system based on deep learning according to claim 1, wherein, Verifying the sensitive area includes verifying the physical space consistency of the sensitive area by correlating the feature mutations of visual data and millimeter-wave radar data.

3. An in-vehicle monitoring system based on deep learning according to claim 1, characterized in that, Collect visual data and millimeter-wave radar data, and predict the space-time distribution pattern based on the motion parameters in the millimeter-wave radar data, including: Synchronously collect visual data and millimeter-wave radar data through a hardware synchronization trigger signal and perform preprocessing; Decompose the motion parameters in the millimeter-wave radar data into the acceleration change rate and the motion direction offset; Calculate the phase offset in the time dimension based on the acceleration change rate, and calculate the projection deviation in the space dimension based on the motion direction offset; Perform time-frequency analysis on the phase offset and the projection deviation to extract the high-energy fluctuation interval in the frequency domain features; Generate a space-time distribution pattern including continuous error and mutation error classification according to the geometric projection relationship between the high-energy fluctuation interval and the sensor installation position.

4. The on-vehicle monitoring system based on deep learning according to claim 1, wherein Dynamically generate the space-time baseline compensation coefficients for visual data and millimeter-wave radar data according to the space-time distribution pattern, including: Calculate the time compensation component based on the phase offset in the time dimension, where the time compensation component is the product of the phase offset and the sampling interval; Calculate the space compensation component based on the projection deviation in the space dimension, where the space compensation component is the ratio of the projection deviation to the sensor installation spacing; Linearly weighted fuse the time compensation component and the space compensation component to generate the space-time baseline compensation coefficient, and the weights of the linear weighted fusion are dynamically allocated according to the ratio of continuous error and mutation error in the space-time distribution pattern.

5. The on-vehicle monitoring system based on deep learning according to claim 1, characterized in that, Construct a space-time correlation model for multi-modal data streams based on the space-time baseline compensation coefficients, and identify sensitive areas in high-speed scenarios, including: Adjust the time interpolation step and the space coordinate transformation parameters of the multi-modal data stream based on the space-time baseline compensation coefficients; Perform time-domain filtering on the speed mutation features in the millimeter-wave radar data to extract candidate sensitive areas where the speed change rate exceeds the dynamic threshold; Map the radar coordinates of the candidate sensitive areas to the visual sensor image plane through the space coordinate transformation parameters to generate the corresponding pixel coordinate area; Extract the edge gradient mutation features of the mapped pixel region in the visual data, and mark the region where the edge gradient magnitude exceeds the preset gradient threshold as the sensitive region.

6. The vehicle-mounted monitoring system based on deep learning according to claim 5, characterized in that, The time interpolation step size is generated by the time compensation component and the preset interpolation window length, and the spatial coordinate transformation parameter is generated by associating the spatial compensation component with the visual sensor resolution.

7. The on-vehicle monitoring system based on deep learning according to claim 2, characterized in that, Verify the physical space consistency of the sensitive region by associating the feature mutations of the visual data and the millimeter-wave radar data. Specifically: Count the number of pixel points whose edge gradient magnitude exceeds the preset gradient threshold; Calculate the spatial distribution matching degree between the velocity mutation magnitude of the radar sensitive region and the number of edge gradient pixel points in the visual sensitive region; If the spatial distribution matching degree exceeds the preset matching threshold and the overlap degree between the pixel coordinate region and the visual sensitive region exceeds the preset ratio, it is determined that the physical space is consistent and the sensitive region verification passes; Otherwise, it is marked as an invalid region and excluded, and the sensitive region verification fails.

8. An in-vehicle monitoring system based on deep learning according to claim 1, characterized in that, For the verified sensitive regions, eliminate the false sensitive regions according to the historical trajectory data, and screen the high-confidence feature segments based on the correlation between the visual features and the millimeter-wave radar velocity distribution, including: Based on the continuity of the target acceleration change rate in the historical trajectory data, construct a physically feasible interval for the acceleration change rate; Eliminate the false sensitive regions where the acceleration change rate exceeds the physically feasible interval. At the same time, perform multi-frame trajectory smoothness verification on the sensitive regions where the acceleration change rate is at the interval boundary. If the number of acceleration direction mutations exceeds the preset number within several consecutive frames, it is determined as a false region; Extract the visual feature segments of the verified sensitive regions, calculate the spatial covariance between the corresponding edge gradient direction distribution and the millimeter-wave radar velocity direction distribution, and extract the covariance reference value in the same type of scene in the historical trajectory at the same time; Dynamically allocate the fusion weight according to the deviation degree between the current spatial covariance and the covariance reference value. The smaller the deviation degree, the higher the visual feature weight, and the larger the deviation degree, the higher the millimeter-wave radar velocity weight; Perform two-way correction on the spatio-temporal features of vision and radar in the multi-modal feature segments. The visual features correct the spatial positioning jumps in the radar velocity distribution, and the radar velocity corrects the motion blur artifacts in the visual features to generate spatio-temporally consistent high-confidence feature segments.

9. The vehicle-mounted monitoring system based on deep learning according to claim 1, wherein Decouple the high-confidence feature segments to suppress the visual motion artifacts and correct the spatial positioning jumps in the millimeter-wave radar data, and generate aligned multi-modal feature vectors, including: Decompose the high-confidence feature segments based on the spatio-temporal dimension, and separate the motion artifact components in the visual features and the positioning jump components in the millimeter-wave radar features; Perform gradient direction filtering on the motion artifact components to suppress the gradient components whose deviation from the radar velocity distribution direction exceeds the preset angle; Perform spatial continuity interpolation on the positioning jump components, and the interpolation step size is dynamically adjusted according to the consistency of the target displacement between adjacent frames in the visual features; Weightedly splice the filtered visual features and the interpolated radar features according to the spatio-temporal baseline compensation coefficient to generate channel-aligned multi-modal feature vectors; Normalize the energy calibration of the multi-modal feature vectors based on the channel weight distribution in the same type of scene in the historical trajectory.

10. A vehicle-mounted monitoring system based on deep learning according to claim 1, characterized in that, Construct a trajectory prediction model by dynamically allocating fusion weights based on multi-modal feature vectors, and inversely correct the spatio-temporal baseline compensation coefficient through the residual error between the predicted trajectory and the measured trajectory, including: Based on the channel weight distribution of historical multi-modal feature vectors, dynamically allocate fusion weights to construct a trajectory prediction model, and the weight allocation rule is generated according to the historical scenario clustering results and the current feature deviation degree; Extract the residual error between the predicted trajectory and the measured trajectory, and the residual error includes horizontal displacement deviation, direction angle deviation and speed change rate difference; Conduct spatio-temporal dimension covariance analysis on the residual error to separate the systematic error components caused by the spatio-temporal baseline compensation coefficient deviation; According to the amplitude direction of the systematic error components, inversely adjust the time compensation component and the space compensation component in the spatio-temporal baseline compensation coefficient; Iteratively input the corrected spatio-temporal baseline compensation coefficient into the spatio-temporal correlation model to update the multi-modal data alignment parameters.

Images