Intelligent driving multi-sensor fusion data processing system

Abstract

The invention discloses an intelligent driving multi-sensor fusion data processing system, which belongs to the technical field of fusion data processing and comprises a data preprocessing unit, a data fusion unit and a data post-processing unit. The data preprocessing unit normalizes the original data of the laser radar, the camera, the millimeter wave radar and the ultrasonic sensor through a standardization, cleaning, noise reduction and calibration synchronization module; the feature level fusion module deeply digs the characteristics of each sensor, and fuses geometric, visual, motion and close-range obstacle features by using a deep neural network; the decision-making level fusion module generates a driving instruction through weighted voting and fuzzy logic reasoning through classification and situation evaluation; and the data post-processing unit combines the vehicle and road condition optimization instruction, evaluates the risk, and stores the data feedback optimization system. The system can improve the sensing precision and decision reliability, enhances the expansion adaptability of the system, and provides guarantee for safe and efficient operation of intelligent driving.

Description

Technical Field

[0001] The present invention relates to the technical field of fusion data processing, and in particular to an intelligent driving multi-sensor fusion data processing system. Background Art

[0002] With the rapid development of intelligent driving technology, multi-sensor fusion data processing has become a core element to improve driving safety and intelligence. In today's complex and ever-changing traffic scenarios, vehicles need to accurately perceive surrounding environmental information and make reliable decisions, which places strict requirements on multi-sensor fusion systems.

[0003] Traditional intelligent driving multi-sensor fusion data processing solutions have many limitations. At the perception level, most early systems simply spliced ​​data collected by different sensors without deeply exploring the inherent correlation of the characteristics of each sensor. For example, lidar data is only roughly used to obtain the general outline of the target, and the camera visual information is not effectively combined with geometric features, resulting in poor target recognition accuracy. At complex intersections and mixed traffic areas, it is common to misidentify targets or fail to accurately locate. It is difficult to clearly distinguish the posture details of different types of vehicles and pedestrians, and it is very easy to miss key road condition information, which lays hidden dangers for driving decisions.

[0004] There are also deficiencies in the decision-making process. In the past, fixed weights were often used to fuse the initial decisions of each sensor, without considering the fluctuations in sensor performance in different scenarios. On sunny days, the weight of the camera's visual information is too high. Once encountering severe weather such as heavy rain and dense fog, the camera is severely disturbed by light and raindrops, but it still participates in decision-making according to the established high weight, resulting in frequent errors in the final driving instructions, causing unnecessary braking and steering misoperations; in the face of fuzzy road conditions, the traditional binary decision-making model lacks a flexible response mechanism, and the judgment criteria of either yes or no cannot match the large number of fuzzy boundary situations in actual driving, and the fault tolerance rate is low, making it difficult to ensure driving safety.

[0005] In addition, the system's scalability and adaptability are the shortcomings of traditional solutions. The old architecture is rigid, and the original design needs to be overturned and the algorithm needs to be reconstructed when connecting to new sensors. The R&D cycle is long and costly. In addition, there is a lack of effective linkage with subsequent data processing links, and it is impossible to optimize sensor parameters and fusion algorithms based on vehicle driving history data and real-time road condition feedback, which makes it difficult for the system to adapt to different vehicle models and road conditions. In special scenarios such as mountain bends and urban congested roads, the response is delayed and the stability is poor, making it difficult to meet the needs of continuous advancement of intelligent driving.

[0006] In summary, the existing intelligent driving multi-sensor fusion data processing technology urgently needs to be improved to overcome the above-mentioned problems, so as to adapt to the increasingly complex and diverse application scenarios of intelligent driving and ensure driving safety and efficient operation of the system. Summary of the invention

[0007] The main purpose of the present invention is to provide an intelligent driving multi-sensor fusion data processing system, which can effectively solve the problems mentioned in the background technology.

[0008] To achieve the above object, the technical solution adopted by the present invention is: An intelligent driving multi-sensor fusion data processing system includes a data preprocessing unit, a data fusion unit and a data post-processing unit: The data preprocessing unit includes a data format standardization module, a data cleaning and noise reduction module, and a data calibration and synchronization module, which are used to convert the format of the received raw data of the lidar, camera, millimeter wave radar, and ultrasonic sensor in a unified manner, eliminate noise and abnormal data, and calibrate and synchronize the spatial coordinates and time.

[0009] The data fusion unit is provided with a feature-level fusion module and a decision-level fusion module. The feature-level fusion module is used to extract corresponding features from the data of each sensor, and adopts a deep neural network architecture to fuse different features into a fusion feature vector that includes target space, vision, motion and close-range information. The decision-level fusion module uses a classifier to carry out fine target classification decisions, and independently performs motion and close-range situation assessments based on millimeter-wave radar and ultrasonic sensor data, and then uses weighted voting and fuzzy logic reasoning strategies to fuse preliminary decision results to generate driving decision instructions.

[0010] The data post-processing unit includes a decision optimization and risk assessment module, a data storage and feedback module, which optimizes driving decision instructions and estimates risk probabilities in combination with the real-time status of the vehicle and road environment information. It is also responsible for storing key data and feedback information to optimize sensor parameters and fusion algorithms, and push warning information on demand.

[0011] The multi-sensor fusion data processing steps of the system are: In the data collection and transmission step, each sensor is started synchronously, collects data at a predetermined frequency and attaches a timestamp, and transmits it to the data preprocessing unit via the high-speed bus in the vehicle.

[0012] In the data preprocessing step, the raw data is processed by the data format standardization module, data cleaning and noise reduction module, and data calibration and synchronization module in turn, and stored in the buffer area waiting for fusion.

[0013] In the data fusion step, the feature-level fusion module extracts the features of each sensor in parallel, completes the deep fusion of multiple features, generates a fused feature vector and sends it to the decision-level fusion module; after receiving the data, the decision-level fusion module first classifies and evaluates it, then integrates the preliminary decision results, and outputs precise driving instructions to the data post-processing unit.

[0014] Data post-processing step: The data post-processing unit optimizes instructions, assesses risks, stores data and feedbacks information, and pushes warnings according to set rules.

[0015] Preferably, the data format standardization module converts the lidar point cloud data into a structured array containing three-dimensional coordinates and reflection intensity information, unifies the camera image data into a digital image matrix with a specific resolution and color encoding format, organizes the millimeter wave radar data into a data set containing target distance, speed, and angle parameters, and processes the ultrasonic sensor data into a simple distance value sequence, and each data is accompanied by a precise timestamp.

[0016] The data cleaning and noise reduction module uses statistical filtering combined with an outlier removal algorithm for laser radar to filter out noise and outlier data based on the local point cloud density threshold, distance threshold and point cloud distribution model; median filtering, Gaussian filtering and image restoration algorithms are used for camera images to remove salt and pepper noise, high-frequency noise and image defects; millimeter-wave radar uses a constant false alarm rate (CFAR) detection algorithm to filter false echoes; and the ultrasonic sensor calibrates the distance measurement value based on the temperature compensation model to eliminate measurement errors caused by temperature fluctuations.

[0017] The data calibration and synchronization module constructs a coordinate transformation matrix, and maps the data of each sensor to the vehicle coordinate system according to the pre-calibrated sensor installation position and angle parameters; uses a high-precision atomic clock or an on-board synchronous bus signal to synchronize the data with a timestamp, and eliminates the time deviation caused by acquisition frequency differences and transmission delays through linear interpolation and timestamp realignment technology.

[0018] Preferably, when the feature-level fusion module extracts the laser radar geometric features, it uses a point cloud segmentation algorithm to divide the target object point cloud cluster, calculates the target bounding box size, and obtains the size parameters by finding the coordinate extreme value along the coordinate axis; calculates the volume by using a convex hull algorithm to decompose the convex hull into a triangular pyramid cumulative volume; calculates the surface flatness, and the neighboring points of the selected point are fitted to the local plane using principal component analysis, and the standard deviation of the distance from the point to the plane is calculated; the centroid position is calculated according to the coordinate mean formula; the specific extraction method is: Point cloud segmentation: The Euclidean clustering algorithm is used for point cloud segmentation. For each point in the point cloud dataset , calculate the Euclidean distance between it and other points ; Set distance threshold , if point With point Distance , they are classified into the same cluster; traverse the entire point cloud dataset and gradually form different clusters, each cluster representing a point cloud cluster of a potential target object.

[0019] Target bounding box size calculation: For each segmented target point cloud cluster, find the maximum value of the point cloud coordinates along the x, y, and z coordinate axes respectively. , , and minimum value , , ; The target bounding box size parameter is , and its size information can intuitively reflect the scope of the target in three-dimensional space.

[0020] Volume calculation: Assuming that the target object is approximately in the shape of a convex hull, the convex hull is constructed using the QuickHull algorithm. The vertex set of the convex hull is , decompose the convex hull into multiple simple triangular pyramids, and use the triangular pyramid volume formula Calculate the volume of each triangular pyramid, where is the area of ​​the base triangle, is high, and the total volume of the target object is accumulated.

[0021] Surface flatness calculation: Select each point in the target point cloud cluster Neighbors ( The value is determined based on experience or experiment, such as ), use the principal component analysis (PCA) method to fit the local plane; calculate the standard deviation of the distance from these points to the fitted plane , The smaller the value, the smoother the surface; otherwise, the rougher the surface.

[0022] Center of mass position calculation: Center of mass coordinates Calculated by the following formula: , , ,in is the number of points in the target point cloud cluster, , , are the coordinates of each point in the point cloud.

[0023] Preferably, the feature-level fusion module extracts the visual features of the camera using a deep learning target detection model, generates a feature map through convolution and downsampling, uses the anchor frame to match the target to predict the category probability, the bounding box coordinate offset and the confidence score, and then retains the high-quality detection results through non-maximum suppression, further convolution and full connection processing are performed on the reserved area to output the shape, texture, color features and category probability to form a visual feature vector. The specific extraction method is: The YOLOv5 model is used for target detection and visual feature extraction. After the image is input into the YOLOv5 network, it passes through several convolutional layers and downsampling layers to generate feature maps of different scales. On the feature map, the pre-set anchor boxes are used to match the target object. Each anchor box predicts the category probability. (covering common categories such as cars, trucks, pedestrians, bicycles, etc.), bounding box coordinate offsets ( , , , ) and target confidence score ,The non-maximum suppression (NMS) algorithm is used to remove overlapping and redundant detection frames and retain high-quality detection results.

[0024] For the retained target detection frame area, visual features are further extracted, and the corresponding area image is processed by convolution layer and fully connected layer to output shape features (such as target aspect ratio, contour curve information), texture features (using gray level co-occurrence matrix GLCM to count the gray level changes of pixels in different directions and distances), color features (extracting RGB channel mean and variance information), together with category probability information to form a visual feature vector for subsequent fusion.

[0025] Preferably, the feature-level fusion module extracts millimeter-wave radar motion features, collects distance, radial velocity, and angle information at fixed time intervals, obtains target radial acceleration by differential operation, calculates angular velocity using two frames of angle information, and combines them into a motion feature vector form. The specific extraction method is: Millimeter wave radar continuously outputs the distance of the target relative to the radar , radial velocity and angle Information at fixed intervals Collect data sequence, target radial acceleration Calculated by difference operation: ,in is the radial velocity at the current moment, is the radial velocity at the previous moment.

[0026] Target angular velocity Calculated by two consecutive frames of angle information: , reflecting the speed of the target's motion direction change; these motion features are combined into a vector form , used to describe the dynamic characteristics of the target.

[0027] Preferably, the feature-level fusion module extracts the close-range obstacle features of the ultrasonic sensor, sets the distance threshold classification, and generates a feature value representing the close-range target state in a binary or graded form based on the comparison between the measured distance and the threshold.

[0028] Preferably, when the feature-level fusion module fuses features, a fused convolutional neural network is constructed, and the laser radar and camera feature branches are respectively subjected to convolution, pooling, and full connection operations. The convolution uses a specific size convolution kernel, step size, and padding method, and the pooling is the maximum pooling. Then, the channels are spliced ​​to output joint features; a recurrent neural network branch is introduced for millimeter wave radar and ultrasonic features, and a long short-term memory unit is used to capture dynamic changes, and finally the outputs of each branch are spliced ​​into a fused feature vector; the specific implementation method is as follows: Fusion convolutional neural network construction and training: Construct a multi-branch convolutional neural network architecture, which is divided into a lidar geometric feature branch and a camera visual feature branch. The lidar geometric feature vector is used as input to a sub-network containing multiple layers of convolution, pooling, and fully connected layers; the camera visual feature vector is also connected to another independent but similar sub-network; in the convolution layer, the Convolution kernel, step size is 1, filling method is SAME, feature extraction is performed, and the calculation formula is: ,in It is Layer feature map, is the convolution kernel weight, It is Layer input features.

[0029] The pooling layer uses maximum pooling, and the window size is , with a step size of 2, the feature map is downsampled to reduce the amount of data and retain key information; after being processed by their respective sub-networks, the feature maps output by the two branches are spliced ​​in the channel dimension and then connected to the subsequent fully connected layer. The number of neurons in the fully connected layer is adjusted according to the complexity of the experiment and task, such as setting it to 256 neurons, and finally outputting a joint feature representation vector to reflect the target space and visual fusion characteristics.

[0030] The recurrent neural network branch processes motion and close-range features: Millimeter wave radar motion feature vector The close-range obstacle characteristics of ultrasonic waves are introduced into the recurrent neural network branch composed of long short-term memory units. In the long short-term memory unit, the core calculation steps are as follows: Forget Gate: ,in is the forget gate weight matrix, is the hidden state at the previous moment, is the current millimeter wave radar motion feature vector, is the current ultrasonic short-range obstacle feature vector, is the bias vector, is the sigmoid function.

[0031] Input Gate: , controls the degree to which the current input information is integrated into the memory unit.

[0032] Update memory unit: , , to realize memory unit update.

[0033] Output Gate: , , determine the current output hidden state, capture the dynamic changes of motion and close-range states over time, and output motion and close-range fusion feature directions after being processed by multiple layers of long short-term memory units.

[0034] Generate fusion feature vector: The spatial-visual fusion feature vector output by the convolutional neural network branch is concatenated with the motion and close-range fusion feature vector output by the recurrent neural network branch in terms of dimension to form a final fusion feature vector, whose dimension is the sum of the output dimensions of the two branches. This vector completely encompasses the target space, vision, motion and close-range information, and is delivered to the decision-level fusion module.

[0035] Preferably, when the decision-level fusion module performs target classification and situation assessment, a support vector machine is used to classify the target, an SVM model is constructed to find the optimal classification hyperplane, the objective function and constraints are set, and the Lagrange multiplier method is used to solve the dual problem to obtain the optimal solution, and the decision function value of the newly input fusion feature vector is calculated to determine the category attribution, specifically including: Target fine classification based on SVM and MLP: Support vector machine (SVM) is used for target classification, and the fused feature vector obtained by feature-level fusion is used as input. For binary classification problems (such as distinguishing between vehicles and pedestrians), the SVM model is constructed to find the optimal classification hyperplane, and the objective function is: , the constraints are , ,in is the hyperplane normal vector, is the bias, is the fusion feature vector, is the category label. The Lagrange multiplier method is used to solve the dual problem and obtain the optimal solution and For the newly input fused feature vector , by calculating the decision function value , and determine the category based on its positive or negative value.

[0036] Multilayer Perceptron (MLP) is used for multi-classification scenario expansion. The network includes an input layer, multiple hidden layers (the number of neurons in the hidden layer can be set to a decreasing structure such as 128, 64, etc.) and an output layer. The activation function uses the ReLU function, and the output layer uses the softmax function to convert the output into the probability distribution of each category, such as ,in is the output layer weight matrix, is the output of the last hidden layer, is the output layer bias, and the category with the highest probability is the target classification result.

[0037] Situation assessment based on millimeter wave radar and ultrasonic sensors: Set speed threshold based on millimeter wave radar data , and acceleration threshold , If the target radial velocity And the acceleration , the target is judged to be approaching at high speed, and the danger level is recorded as high; if and , the danger level is recorded as medium; if and , the danger level is recorded as low.

[0038] Combined with the close-range obstacle characteristics of the ultrasonic sensor, when the characteristic value is 2 (extremely close), the overall danger level is directly increased; if it is 1 (close) and the millimeter-wave radar also indicates that the target is approaching, the danger level is also adjusted to a higher state, and a comprehensive situation assessment result is formed for subsequent decision fusion.

[0039] Preferably, when the decision-level fusion module generates the final driving decision through fusion, weighted voting is used to fuse the preliminary decision to dynamically assign weights to each sensor according to different scenarios, and the driving operation is determined by weighted summation of the comprehensive score; when fuzzy logic reasoning is used, fuzzy sets and membership functions are defined, a fuzzy rule base is formulated, and driving instructions are output through fuzzy synthesis and centroid method defuzzification, specifically including: Weighted Voting Method: Assign weights to each sensor based on different scenarios. For example, in a strong light scene, the reliability of the camera is reduced due to light interference, and the weight is set to ; The laser radar weight is set to ; Millimeter wave radar is less affected by light, and the weight is set to ;The ultrasonic sensor weight is set to When the SVM classification target is a pedestrian and the confidence level is 0.8 (denoted as ), the millimeter-wave radar situation assessment danger level is medium (denoted as ), the ultrasonic sensor indicates that there is an obstacle in the close distance (denoted as ), weighted voting comprehensive score . According to the predefined threshold, if , output a deceleration command; if it is between 0.3 - 0.5, maintain the current vehicle speed and strengthen monitoring; if it is less than 0.3, drive normally.

[0040] Fuzzy logic reasoning: Define fuzzy sets and membership functions. For target distance, set the fuzzy sets of "near", "medium", and "far", for speed, set the fuzzy sets of "slow", "medium", and "fast", and for danger level, set the fuzzy sets of "high", "medium", and "low". For example, for target distance, use the triangular membership function. If the actual distance of the target is , the membership function of the “near” fuzzy set is: ,in , It is a distance parameter, which is set according to the actual scene.

[0041] Formulate a fuzzy rule base, such as "IF the laser radar target is close AND the camera identifies it as a pedestrian AND the millimeter wave radar detects it quickly AND the ultrasonic wave indicates that there is an obstacle at close range THEN the danger level is high". Calculate the membership value of the fuzzy set based on the data of each sensor, obtain the fuzzy output result through fuzzy synthesis operation (such as Mamdani reasoning method), and finally use the center of gravity method to defuzzify and convert the fuzzy result into a clear driving decision instruction. Assuming that the fuzzy output danger level membership distribution is [0.2 (low), 0.3 (medium), 0.5 (high)], the corresponding driving decision strength values ​​are [0 (no operation), 0.5 (deceleration), 1 (emergency braking)], and the defuzzification calculation formula is: , calculated , and outputs corresponding driving decision instructions, such as deceleration or braking operations.

[0042] Preferably, the data post-processing unit comprises a decision optimization and risk assessment module and a data storage and feedback module, wherein: The decision optimization and risk assessment module is used to optimize and adjust the fusion decision instructions based on the vehicle's real-time speed, acceleration, steering angle, tire grip information and the road curvature, slope, lane width, and traffic rules information of the road environment. When a deceleration instruction is received while driving at high speed, the braking deceleration curve is optimized based on the vehicle speed and vehicle dynamics model to prevent sudden braking and loss of control. When turning, the steering angle instruction is accurately fine-tuned based on the road curvature to ensure smooth cornering. At the same time, based on historical data and current conditions, the probability of potential collision and lane deviation risks is predicted to adjust the driving strategy in advance.

[0043] The data storage and feedback module is responsible for storing the original sensor data, preprocessing results, fusion feature vectors, decision-making process, and vehicle driving status information in a time series to the local database for accident backtracking, algorithm performance evaluation, and system troubleshooting; regularly statistically analyzing the stored data, mining sensor performance trends and data fusion effects, adjusting sensor parameters and optimizing fusion algorithms based on the feedback of the results, and realizing system adaptive upgrades.

[0044] Compared with the prior art, the present invention has the following beneficial effects: 1. The feature-level fusion module deeply mines the characteristics of each sensor, comprehensively extracts the characteristics of laser radar geometry, camera vision, millimeter-wave radar motion, and ultrasonic sensor close-range obstacles, abandons the traditional simple fusion method, and uses the deep neural network architecture to make various features fully complementary and deeply intertwined. In complex traffic scenes, such as busy intersections and expressway entrances and exits where vehicles frequently change lanes, it can accurately outline the contours of surrounding targets and capture their dynamic changes in detail, greatly reducing missed judgments and misjudgments caused by fuzzy and one-sided perception, laying a solid foundation for accurate perception for subsequent decision-making, and allowing the intelligent driving system to clearly understand the surrounding conditions.

[0045] 2. The decision-level fusion module first uses classifiers and data evaluation based on millimeter-wave radar and ultrasonic sensors to achieve multi-dimensional and accurate analysis of road conditions; the weighted voting strategy flexibly adjusts the weight of each sensor according to the actual scenario, highlights the advantages of millimeter-wave radar in bad weather, and focuses on lidar data in strong light environments, ensuring that decisions are in line with current road conditions and effectively avoiding unreasonable instructions, greatly enhancing the accuracy and reliability of decisions; the fuzzy logic reasoning system relies on customized fuzzy sets and a rich rule base to handle common fuzzy and uncertain situations in driving, output continuous and reasonable flexible decision instructions, avoid decisions falling into the limitations of black and white, and calmly respond to sudden or complex road conditions.

[0046] 3. From the perspective of system architecture, the data fusion unit is designed with significant modular and structured features. If new sensors are to be connected in the future, they can be efficiently integrated by referring to the existing feature extraction and fusion processes, which is conducive to rapid iteration and upgrading, saving R&D costs and time. Furthermore, it is closely linked with the data post-processing unit. The post-processing unit deeply analyzes the performance trends of sensors and the effectiveness of data fusion based on the massive amount of stored data, and provides accurate feedback based on this, dynamically adjusts sensor parameters, and iterates fusion algorithms, so that the system's response speed is greatly improved under different vehicle models and various road conditions, and its adaptability is significantly enhanced, which can keep up with the changing needs of intelligent driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the composition architecture flow of the present invention. DETAILED DESCRIPTION

[0048] The following will describe the implementation of the present invention in detail with reference to the accompanying drawings and examples.

[0049] like Figure 1 As shown, an intelligent driving multi-sensor fusion data processing system is specifically divided into three parts: a data preprocessing unit, a data fusion unit and a data post-processing unit.

[0050] In actual applications, when a smart driving vehicle is started, the laser radar, camera, millimeter wave radar and ultrasonic sensor start working simultaneously. The laser radar quickly scans the vehicle's surrounding environment at a frequency of 15Hz, emitting laser beams and receiving reflected light to form point cloud data containing rich three-dimensional spatial information. In parking lot scenarios, the laser radar can clearly capture the outline and spacing of surrounding parked vehicles, as well as the location information of fixed obstacles such as pillars and walls, providing basic data for vehicle starting and parking planning; the camera captures high-definition images of the vehicle's surroundings at a frame rate of 30fps, covering the front, rear, and surround views. The front camera focuses on identifying lane lines, traffic signs, vehicles in front, and pedestrians, just like the driver's "eyes" to obtain visual scene information in real time; the millimeter-wave radar continuously emits millimeter-wave signals, collecting the distance, radial velocity, and angle information of the target relative to the radar at intervals of 0.1s, and is good at monitoring long-distance high-speed moving targets. In high-speed driving scenarios, it can lock in advance on vehicles that are approaching or moving away quickly several meters away, and assist in judging the safe distance between vehicles; the ultrasonic sensor detects the close-range area around the vehicle at a high frequency of 20Hz, accurately locates obstacles within a range of 0-2 meters, and closely monitors the distance between the four corners of the vehicle body and surrounding objects when the vehicle is moving at low speed or entering the warehouse to prevent scratches. Each sensor attaches a precise timestamp to the collected data, which is quickly transmitted to the data preprocessing unit via the high-speed CAN bus in the vehicle.

[0051] 1. Data Preprocessing 1. Data format standardization The data format standardization module quickly started working. The laser radar point cloud data was parsed and converted into a structured array format, which recorded the three-dimensional coordinates and reflection intensity information of each point cloud in detail, facilitating the direct call and operation of subsequent algorithms; the camera image data was uniformly adjusted to a digital image matrix with a resolution of 1920×1080 and RGB888 color encoding format, which met the input requirements of the deep learning model; the millimeter wave radar sorted the collected scattered signal data into a regular data set, clearly listing the distance, speed, and angle values ​​of each target; the ultrasonic sensor data was simplified into a simple distance value sequence, and the timestamps carried by each data were calibrated and synchronized with other sensors, and stored in a dedicated cache area, ready for cleaning and noise reduction processing.

[0052] 2. Data cleaning and noise reduction For the lidar data, the statistical filtering algorithm is started, and the local point cloud density threshold and distance threshold are set to filter out isolated noise points and outlier data caused by abnormal reflection. For example, in a scene with strong direct light, some laser beams are reflected too strongly to form false far points, which can be accurately removed by this algorithm; the camera image uses median filtering to remove salt and pepper noise, the window size is set to 3×3, the image details are smoothed, and then combined with Gaussian filtering to suppress high-frequency noise. At the same time, the image repair algorithm repairs bad points and defects caused by lens dirt and glare; the millimeter wave radar uses the constant false alarm rate (CFAR) detection algorithm to filter out false echoes generated by multipath reflection and electromagnetic interference with a false alarm probability of 1%; the ultrasonic sensor uses the real-time temperature value collected by the built-in temperature sensor, substitutes it into the temperature compensation formula to calibrate the distance value, eliminates the measurement error caused by temperature fluctuations, improves data purity, and the processed data flows back to the cache area.

[0053] 3. Data calibration and synchronization The coordinate transformation matrix is ​​constructed based on the sensor installation position and angle parameters that have been precisely calibrated in advance. Assuming that the laser radar is installed in the center of the front roof, with an angle of 0 degrees and a height of 1 meter with the vehicle's central axis, the point cloud data is mapped to the vehicle coordinate system based on this; the high-precision on-board atomic clock signal is used to synchronize the sensor data with the timestamp, and linear interpolation technology is used to adjust the time deviation caused by the difference in acquisition frequency and transmission delay. For example, if the laser radar data at a certain moment lags behind for a short period of time due to transmission delay, the data of this period is supplemented by linear interpolation, so that the data of each sensor is accurately synchronized in the time dimension, ensuring that the data collected at the same time participates in the fusion processing.

[0054] 2. Data Fusion 1. Feature-level fusion Feature extraction: The laser radar geometric feature extraction uses the Euclidean clustering algorithm for point cloud segmentation. For any point in the point cloud data set, calculate its Euclidean distance to the surrounding points, set a suitable distance threshold, and if the distance between two points is less than the threshold, they are classified into the same cluster, and traverse the data set to form point cloud clusters of different target objects. When calculating the size of the target bounding box, find the extreme values ​​of the coordinates along the coordinate axis to accurately present the spatial occupancy range of targets such as vehicles; use the QuickHull algorithm to calculate the target volume, first construct a convex hull and then decompose it into a triangular pyramid cumulative volume; select a certain number of neighboring points for each point to fit the local plane using principal component analysis (PCA), and calculate the standard deviation of the point-to-plane distance to evaluate the surface flatness; calculate the center of mass position according to the coordinate mean formula to locate the center of gravity of the target.

[0055] The YOLOv5 model is used for camera visual feature extraction. The image is input into the network, and feature maps of different scales are generated through multiple convolutional layers and downsampling layers. The anchor frame is used to match the target, predict the category probability (covering common categories), bounding box coordinate offset and confidence score, and remove redundant detection frames through non-maximum suppression. For the reserved area, it is further processed by convolution and fully connected layers to output shape features (analyzing aspect ratio and contour curve), texture features (using gray-level co-occurrence matrix GLCM to count pixel grayscale changes), and color features (extracting RGB channel mean and variance) to form a visual feature vector.

[0056] Millimeter wave radar motion feature extraction collects data at fixed time intervals. The target radial acceleration is obtained through differential operation, and the angular velocity is calculated from the angle information of two consecutive frames, which are combined into a motion feature vector to describe the target dynamics.

[0057] The ultrasonic sensor extracts close-range obstacle features, sets thresholds such as very close, close, and relatively close, generates feature values ​​based on the measured distance comparison, and intuitively reflects the state of close-range targets.

[0058] Feature Fusion: Construct a fusion convolutional neural network (CNN), and the feature branches of the lidar and camera are each subjected to multi-layer convolution, pooling, and full connection operations. The convolution layer uses a convolution kernel of appropriate size, and the step size and padding method are carefully designed to extract features; the pooling layer performs downsampling to reduce the amount of data; the output feature maps of the two branches are spliced ​​in the channel dimension, connected to a fully connected layer with a specific number of neurons, and the joint feature representation vector is output.

[0059] A recurrent neural network (RNN) branch is introduced for millimeter-wave radar and ultrasonic features, and a long short-term memory unit (LSTM) is used. The forget gate controls memory retention; the input gate adjusts information integration; the memory unit is updated to achieve dynamic storage; the output gate outputs the hidden state, and after multi-layer processing, the motion and close-range fusion feature vectors are output; finally, the two branch outputs are spliced ​​to form a fusion feature vector that includes all-round information of the target and is sent to the decision-level fusion module.

[0060] 2. Decision-level integration Target classification and situation assessment: Support vector machine (SVM) is used to classify targets, and an SVM model is built to find the optimal classification hyperplane. The objective function and constraints are set, and the Lagrange multiplier method is used to solve the dual problem to obtain the optimal solution. Based on this, the category of the newly input fused feature vector is determined. At the same time, based on the millimeter-wave radar and ultrasonic sensor data, the speed threshold, acceleration threshold and ultrasonic distance threshold are set to evaluate the target motion and close-range situation.

[0061] Fusion generates the final driving decision: Weighted voting fusion makes a preliminary decision. The weights of the camera in the strong light direct scene are set to 0.2, the laser radar to 0.3, the millimeter wave radar to 0.4, and the ultrasonic sensor to 0.1. If the SVM classification target is a pedestrian and the confidence is high, the millimeter wave radar prompts the target to approach at a low speed, and the ultrasonic sensor detects obstacles at close range. The weighted sum comprehensive score determines the driving operation. Fuzzy logic reasoning defines fuzzy sets (such as the target distance is set to "near", "medium" and "far" fuzzy sets), membership functions, and formulates a fuzzy rule base (such as "IF the laser radar target is close and large, the camera identifies it as a truck, the millimeter wave radar detects quickly, and the ultrasonic sensor prompts that there is an obstacle at close range THEN the danger level is high"), and outputs driving instructions such as acceleration, deceleration, steering, braking, etc. through fuzzy synthesis and defuzzification by the center of gravity method to accurately command vehicle actions.

[0062] 3. Data Post-Processing 1. Decision optimization and risk assessment When the vehicle receives a deceleration command while driving at high speed, the decision optimization and risk assessment module optimizes the braking deceleration curve based on the vehicle speed (such as 120km / h) and the vehicle dynamics model to avoid sudden braking and loss of control. In turning scenarios, the steering angle command is accurately fine-tuned according to the road curvature (such as a curve radius of 50 meters) to ensure smooth cornering. At the same time, based on historical data (past records of similar road conditions) and current situations (real-time sensor fusion data), the module predicts the probability of potential collision and lane deviation risks, and adjusts driving strategies in advance, such as early warning, fine-tuning the vehicle speed, or maintaining a wider distance between vehicles.

[0063] 2. Data storage and feedback The data storage and feedback module stores key information such as raw sensor data, preprocessing results, fusion feature vectors, decision-making process, vehicle driving status, etc. in a local database in time series for accident backtracking, algorithm performance evaluation, and system troubleshooting; regularly statistically analyzes stored data to mine sensor performance trends and data fusion effects. If it is found that the laser radar point cloud noise increases during a certain period of time, feedback is provided to adjust its transmission power and scanning frequency; the algorithm parameters are optimized based on the fusion effect to achieve system adaptive upgrades and continuously improve intelligent driving performance and safety.

[0064] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An intelligent driving multi-sensor fusion data processing system, characterized by: Including data pre-processing unit, data fusion unit and data post-processing unit: The data preprocessing unit includes a data format standardization module, a data cleaning and noise reduction module, and a data calibration and synchronization module, which are used to perform unified format conversion, noise and abnormal data elimination, and calibration and synchronization of spatial coordinates and time on the raw data received from the laser radar, camera, millimeter wave radar, and ultrasonic sensor in sequence; The data fusion unit is provided with a feature-level fusion module and a decision-level fusion module. The feature-level fusion module is used to extract corresponding features from each sensor data, and adopts a deep neural network architecture to fuse different features into a fusion feature vector that includes target space, vision, motion and close-range information. The decision-level fusion module uses a classifier to carry out fine classification decisions of targets, and independently performs motion and close-range situation assessment based on millimeter-wave radar and ultrasonic sensor data, and then uses weighted voting and fuzzy logic reasoning strategies to fuse preliminary decision results to generate driving decision instructions. The data post-processing unit includes a decision optimization and risk assessment module, a data storage and feedback module, which optimizes driving decision instructions and estimates risk probability in combination with the real-time status of the vehicle and road environment information. It is also responsible for storing key data and feedback information to optimize sensor parameters and fusion algorithms, and push warning information on demand; The multi-sensor fusion data processing steps of the system are: In the data collection and transmission step, each sensor is started synchronously, collects data at a predetermined frequency and attaches a time stamp, and transmits the data to the data preprocessing unit via the high-speed bus in the vehicle; In the data preprocessing step, the raw data is processed by the data format standardization module, the data cleaning and noise reduction module, and the data calibration and synchronization module in turn, and then stored in the buffer area to wait for fusion; In the data fusion step, the feature-level fusion module extracts the features of each sensor in parallel, completes the deep fusion of multiple features, generates a fusion feature vector and sends it to the decision-level fusion module; after receiving it, the decision-level fusion module first classifies and evaluates it, then integrates the preliminary decision results, and outputs precise driving instructions to the data post-processing unit; Data post-processing steps: The data post-processing unit optimizes instructions, assesses risks, stores data and feedbacks information, and pushes warnings according to set rules.

2. The intelligent driving multi-sensor fusion data processing system according to claim 1, characterized in that: The data format standardization module converts the laser radar point cloud data into a structured array containing three-dimensional coordinates and reflection intensity information, unifies the camera image data into a digital image matrix with a specific resolution and color coding format, organizes the millimeter wave radar data into a data set containing target distance, speed, and angle parameters, and processes the ultrasonic sensor data into a simple distance value sequence, and each data is accompanied by a precise timestamp; The data cleaning and noise reduction module uses statistical filtering combined with an outlier removal algorithm for laser radar to filter out noise and outlier data based on the local point cloud density threshold, distance threshold and point cloud distribution model; Median filtering, Gaussian filtering and image restoration algorithms are used on camera images to remove salt and pepper noise, high-frequency noise and image defects; millimeter-wave radar uses a constant false alarm rate detection algorithm to filter false echoes; ultrasonic sensors calibrate the distance value based on the temperature compensation model to eliminate measurement errors caused by temperature fluctuations; The data calibration and synchronization module constructs a coordinate transformation matrix, and maps the data of each sensor to the vehicle coordinate system according to the pre-calibrated sensor installation position and angle parameters; uses a high-precision atomic clock or an on-board synchronous bus signal to synchronize the data with a timestamp, and eliminates the time deviation caused by acquisition frequency differences and transmission delays through linear interpolation and timestamp realignment technology.

3. The intelligent driving multi-sensor fusion data processing system according to claim 1, characterized in that: When the feature-level fusion module extracts the laser radar geometric features, it uses the point cloud segmentation algorithm to divide the target object point cloud cluster, calculates the target bounding box size, and obtains the size parameters by finding the coordinate extreme value along the coordinate axis; the volume is calculated by decomposing the convex hull into a triangular pyramid cumulative volume using the convex hull algorithm; the neighboring points of the selected point for surface flatness are fitted to the local plane using principal component analysis, and the standard deviation of the distance from the point to the plane is calculated; the centroid position is calculated according to the coordinate mean formula; the specific extraction method is: Point cloud segmentation: The Euclidean clustering algorithm is used for point cloud segmentation. For each point in the point cloud dataset , calculate the Euclidean distance between it and other points ; Set distance threshold , if point With point Distance , they are classified into the same cluster; traverse the entire point cloud data set and gradually form different clusters, each cluster represents a point cloud cluster of a potential target object; Target bounding box size calculation: For each segmented target point cloud cluster, find the maximum value of the point cloud coordinates along the x, y, and z coordinate axes respectively. , , and minimum value , , ; The target bounding box size parameter is , its size information can intuitively reflect the scope occupied by the target in three-dimensional space; Volume calculation: Assuming that the target object is approximately in the shape of a convex hull, the convex hull is constructed using the QuickHull algorithm. The vertex set of the convex hull is , decompose the convex hull into multiple simple triangular pyramids, and use the triangular pyramid volume formula Calculate the volume of each triangular pyramid, where is the area of ​​the base triangle, is high, and the total volume of the target object is obtained by accumulation; Surface flatness calculation: Select each point in the target point cloud cluster Neighboring points, The value is determined based on experience or experiment, and the principal component analysis method is used to fit the local plane; the standard deviation of the distance from these points to the fitting plane is calculated , The smaller the value, the higher the surface flatness; otherwise, the rougher the surface; Center of mass position calculation: Center of mass coordinates Calculated by the following formula: , , ,in is the number of points in the target point cloud cluster, , , are the coordinates of each point in the point cloud.

4. The intelligent driving multi-sensor fusion data processing system according to claim 3 is characterized by: The feature-level fusion module extracts the visual features of the camera using a deep learning target detection model, generates a feature map through convolution and downsampling, uses the anchor frame to match the target to predict the category probability, bounding box coordinate offset and confidence score, and then retains the high-quality detection results through non-maximum suppression. Further convolution and full connection processing are performed on the reserved area to output shape, texture, color features and category probability to form a visual feature vector. The specific extraction method is: The YOLOv5 model is used for target detection and visual feature extraction. After the image is input into the YOLOv5 network, it passes through several convolutional layers and downsampling layers to generate feature maps of different scales. On the feature map, the pre-set anchor box is used to match the target object, and each anchor box predicts the category probability. , bounding box coordinate offset ( , , , ) and target confidence score , remove overlapping and redundant detection frames through the non-maximum suppression algorithm to retain high-quality detection results; For the retained target detection frame area, visual features are further extracted, and the corresponding area image is processed through the convolution layer and the fully connected layer to output shape features, texture features, color features, and category probability information to form a visual feature vector for subsequent fusion.

5. The intelligent driving multi-sensor fusion data processing system according to claim 4, characterized in that: The feature-level fusion module extracts millimeter-wave radar motion features, collects distance, radial velocity, and angle information at predetermined time intervals, obtains target radial acceleration through differential calculation, calculates angular velocity using two frames of angle information, and combines them into a motion feature vector form. The specific extraction method is as follows: Millimeter wave radar continuously outputs the distance of the target relative to the radar , radial velocity and angle Information at fixed intervals Collect data sequence, target radial acceleration Calculated by difference operation: ,in is the radial velocity at the current moment, is the radial velocity at the previous moment; Target angular velocity Calculated by two consecutive frames of angle information: , reflecting the speed of the target's motion direction change; these motion features are combined into a vector form , used to describe the dynamic characteristics of the target.

6. The intelligent driving multi-sensor fusion data processing system according to claim 5, characterized in that: The feature-level fusion module extracts the close-range obstacle features of the ultrasonic sensor, sets the distance threshold classification, and generates a feature value representing the close-range target state in a binary or graded form based on the comparison between the measured distance and the threshold.

7. The intelligent driving multi-sensor fusion data processing system according to claim 6, characterized in that: When the feature-level fusion module fuses features, a fused convolutional neural network is constructed. The laser radar and camera feature branches are each subjected to convolution, pooling, and full connection operations. The convolution uses a specific size convolution kernel, step size, and padding method. The pooling is the maximum pooling, and then the splicing channels output joint features. For millimeter-wave radar and ultrasonic features, a recurrent neural network branch is introduced, and the long short-term memory unit is used to capture dynamic changes. Finally, the outputs of each branch are spliced ​​into a fused feature vector. The specific implementation method is as follows: Fusion convolutional neural network construction and training: Construct a multi-branch convolutional neural network architecture, which is divided into a lidar geometric feature branch and a camera visual feature branch. The lidar geometric feature vector is used as input to a sub-network containing multiple layers of convolution, pooling, and fully connected layers; the camera visual feature vector is also connected to another independent but similar sub-network; in the convolution layer, the Convolution kernel, step size is 1, filling method is SAME, feature extraction is performed, and the calculation formula is: ,in It is Layer feature map, is the convolution kernel weight, It is Layer input features; The pooling layer uses maximum pooling, and the window size is , with a step size of 2, the feature map is downsampled to reduce the amount of data and retain key information; after being processed by their respective sub-networks, the feature maps output by the two branches are spliced ​​in the channel dimension and then connected to the subsequent fully connected layer. The number of neurons in the fully connected layer is adjusted according to the complexity of the experiment and task, and finally the joint feature representation vector is output to reflect the target space and visual fusion characteristics; The recurrent neural network branch processes motion and close-range features: Millimeter wave radar motion feature vector The close-range obstacle characteristics of ultrasonic waves are introduced into the recurrent neural network branch composed of long short-term memory units. In the long short-term memory unit, the core calculation steps are as follows: Forget Gate: ,in is the forget gate weight matrix, is the hidden state at the previous moment, is the current millimeter wave radar motion feature vector, is the current ultrasonic short-range obstacle feature vector, is the bias vector, is the sigmoid function; Input Gate: , controls the degree to which the current input information is integrated into the memory unit; Update memory unit: , , realize memory unit update; Output Gate: , , determine the current output hidden state, capture the dynamic changes of motion and close-range states over time, and output motion and close-range fusion feature vectors after being processed by multiple layers of long short-term memory units; Generate fusion feature vector: The spatial-visual fusion feature vector output by the convolutional neural network branch is concatenated with the motion and close-range fusion feature vector output by the recurrent neural network branch in terms of dimension to form a final fusion feature vector, whose dimension is the sum of the output dimensions of the two branches. This vector completely encompasses the target space, vision, motion and close-range information, and is delivered to the decision-level fusion module.

8. The intelligent driving multi-sensor fusion data processing system according to claim 1, characterized in that: When the decision-level fusion module performs target classification and situation assessment, it adopts support vector machine to classify targets, builds support vector machine model to find the optimal classification hyperplane, sets objective function and constraints, uses Lagrange multiplier method to solve the dual problem and obtains the optimal solution, and calculates the decision function value of the newly input fusion feature vector to determine the category attribution.

9. The intelligent driving multi-sensor fusion data processing system according to claim 1, characterized in that: When the decision-level fusion module generates the final driving decision through fusion, weighted voting is used to fuse the preliminary decision to dynamically assign weights to each sensor according to different scenarios, and the driving operation is determined by the weighted sum of the comprehensive scores; when fuzzy logic reasoning is used, fuzzy sets and membership functions are defined, a fuzzy rule base is formulated, and the driving instructions are output through fuzzy synthesis and defuzzification by the center of gravity method.

10. The intelligent driving multi-sensor fusion data processing system according to claim 1, characterized in that: The data post-processing unit includes a decision optimization and risk assessment module and a data storage and feedback module, wherein: The decision optimization and risk assessment module is used to optimize and adjust the fusion decision instructions by combining the vehicle's real-time speed, acceleration, steering angle, tire grip information, and the road curvature, slope, lane width, and traffic rules information of the road environment; The data storage and feedback module is responsible for storing the original sensor data, preprocessing results, fusion feature vectors, decision-making process, and vehicle driving status information in a time series to the local database for accident backtracking, algorithm performance evaluation, and system troubleshooting; regularly statistically analyzing the stored data, mining sensor performance trends and data fusion effects, adjusting sensor parameters and optimizing fusion algorithms based on the feedback of the results, and realizing system adaptive upgrades.

Images