Intelligent vehicle dynamic target three-dimensional sensing method in complex environment
Through multi-sensor data fusion and deep learning models, intelligent vehicles realize dynamic target capture and route planning in complex environments, solving the problem of inflexible route planning in the existing technology, and improving driving efficiency and safety.
Patent Information
- Application Number
- CN202410962360.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-12-17
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing smart vehicles are difficult to achieve dynamic target capture and flexible route planning in complex environments, resulting in wasted time and energy.
A variety of sensor data fusion is adopted, combined with convolutional neural networks and deep learning models for real-time object detection and classification, a three-dimensional map is built, target behavior is predicted, and vehicle driving trajectory and speed are adjusted through semantic segmentation.
It realizes dynamic target capture and route planning of intelligent vehicles in complex environments, saves time and energy, and improves driving efficiency and safety.
Smart Images

Figure CN119152490A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent vehicle driving, and specifically relates to a three-dimensional perception method of dynamic targets of intelligent vehicles in complex environments. Background Art
[0002] An intelligent vehicle is a comprehensive system that integrates environmental perception, planning and decision-making, and multi-level assisted driving. It concentrates on the use of computers, modern sensors, information fusion, communications, artificial intelligence, and automatic control technologies. It is a typical high-tech complex. The research on intelligent vehicles is mainly committed to improving the safety and comfort of automobiles, as well as providing an excellent human-vehicle interaction interface. In recent years, intelligent vehicles have become a hot topic in the world's vehicle engineering research field and a new driving force for the growth of the automobile industry. Many developed countries have incorporated them into their respective key intelligent transportation systems.
[0003] Existing smart cars cannot achieve dynamic target capture when planning routes. They are too rigid during driving and cannot flexibly select and plan routes, which easily leads to a waste of a lot of time and energy. Summary of the invention
[0004] In view of the deficiencies in the prior art, the present invention provides a three-dimensional perception method for dynamic targets of intelligent vehicles in complex environments, which has the advantages of being able to capture dynamic targets, flexibly select and plan routes during driving, and saving a lot of time and energy.
[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a three-dimensional perception method for dynamic targets of smart vehicles in complex environments, the perception method comprising: acquiring data from cameras, lidars, radars and ultrasonic sensors, and fusing data from multiple sensors; using a deep learning model of a convolutional neural network to perform real-time target detection and classification, and combining the target motion model to track the target, and infer the future position and speed of the target; converting the point cloud data obtained by the lidar into a three-dimensional scene model, combining vision, lidar and inertial measurement unit data to build a three-dimensional map around the vehicle in real time, and locate and track the target therein; using deep learning technology to perform semantic segmentation on the image obtained by the camera, dividing the scene into different semantic areas, combining the target's motion pattern, information about the surrounding environment and historical data, performing situational reasoning, and predicting the target's possible behavior and future trends; transmitting the perceived target information to the decision-making and control system of the smart vehicle, and adjusting the vehicle's driving trajectory and speed in real time.
[0006] Preferably, the step of S1 is:
[0007] Synchronize the data, obtain the timestamp, and calibrate the timestamp;
[0008] Parse the data into a uniform format;
[0009] According to application requirements and sensor capabilities, select the level of data fusion, including sensor-level fusion, feature-level fusion, and decision-level fusion;
[0010] For each sensor, the measurement accuracy and confidence are evaluated through measurement accuracy analysis;
[0011] According to the characteristics and evaluation results of the sensor, the weighted average data fusion algorithm is selected. Different weights are assigned according to the confidence of the sensor. The fused data is obtained by weighted average calculation. The weight is dynamically adjusted according to the measurement accuracy of the sensor. The specific steps are as follows:
[0012] Detailed measurement accuracy and confidence assessment for each sensor:
[0013] Analyze the measurement error range and stability of each sensor under different conditions;
[0014] Considering the reliability, working status and environmental adaptability of the sensor, the confidence assessment is carried out through the verification results and fault analysis in actual operation;
[0015] According to the evaluation results, the weight distribution strategy of each sensor in data fusion is determined, and the weight distribution is based on the measurement accuracy and confidence:
[0016] The data fusion process of performing weighted average method:
[0017] Weighted average calculation: According to the set weight, the measurement value of each sensor is weighted averaged. The calculation formula of weighted average is as follows:
[0018]
[0019] Among them, FV is the fused value, Sen i is the measurement value of the ith sensor, w i is the weight of the i-th sensor, n is the number of sensors;
[0020] The fusion result after weighted average is obtained and used for further decision-making and control of the system.
[0021] Preferably, a data fusion process is performed to integrate information from different sensors into a unified data representation, the specific steps of which include:
[0022] Convert the data from each sensor into a unified data space or coordinate system;
[0023] The final fusion data is used for system decision-making and control, providing accurate environmental perception information;
[0024] In practical applications, the accuracy and real-time performance of fused data are continuously verified, and sensor selection, weight distribution, and fusion strategies are optimized based on actual feedback.
[0025] Preferably, the step of S2 is:
[0026] Get real-time video stream and image sequence as input data, determine the target category and category label, use CNN for target detection and classification, and the CNN classification formula is:
[0027]
[0028] Among them, I i+m,j+n,l· is the element of the input data, W m,n,l,k is the parameter of the kth filter, b k It is k The bias of the filter, σ is the activation function, O ijk is the output of the convolution operation;
[0029] Select the SSD target detection task model;
[0030] Preprocess the input image, including normalization and resizing;
[0031] Extract features from images through convolution and pooling operations;
[0032] Identify and locate objects of interest in images and classify them into predefined categories;
[0033] Output detection box, target category and confidence score;
[0034] Target tracking is performed in conjunction with the target motion model, using the target position information detected in the previous steps as the starting point. The goal of target tracking is to maintain the target's identity between consecutive frames and predict its future position and speed.
[0035] Preferably, a recurrent neural network motion model is used;
[0036] The motion model formula is:
[0037] xt=RNN(xt-1,u t )]
[0038] Among them, RNN represents the transfer function of the recurrent neural network, which receives the state xt-1 at the previous moment and the input u at the current moment t , and then output the current state xt;
[0039] Initialize the tracker using the detection results in the first frame, including the target location and category;
[0040] Predict the target's future position and velocity based on its current position and motion model;
[0041] The position prediction formula is:
[0042] [xt+k=RNN(xt+k-1,ut+k-1)]
[0043] The initial conditions are the position xt and input ut at the current time t;
[0044] The speed prediction formula is:
[0045] [vt = RNN v (v t-1 ,ut)]
[0046] Among them, RNN v It is a RNN transfer function specifically used to predict speed;
[0047] Associate the detection results in the current frame with the tracked target in the previous frame;
[0048] Updates the status of the tracker, including position and velocity.
[0049] Preferably, the step of S3 is:
[0050] LiDAR acquires point cloud data by emitting laser beams and measuring their reflections. Visual sensors provide image data for object detection, feature extraction, and visual positioning. IMU provides the vehicle's accelerometer and gyroscope data. The point cloud data acquired by LiDAR is denoised, filtered, and segmented. Features such as surface normals, planes, edges, etc. are extracted from the point cloud data for subsequent map construction and target detection. LiDAR and visual data are used to initialize the map around the vehicle. Real-time LiDAR data is used to compare and fuse new point cloud data with existing maps.
[0051] Preferably, vision and point cloud data are used for target detection and recognition, data from lidar, vision and IMU are fused, a multi-target tracking algorithm is used to track the detected targets, and the map is continuously updated to ensure that real-time environmental changes such as moving obstacles or new road structures are reflected. The map, vehicle status and estimated target position information are fed back to the vehicle's navigation system or driver assistance system for path planning, obstacle avoidance and navigation decisions.
[0052] Preferably, the step of S4 is:
[0053] Use deep learning technology to perform semantic segmentation on the image, assign each pixel in the image to a different semantic category, use the object detection model in deep learning to identify and locate the position and bounding box of each object in the semantically segmented image, and track the movement trajectory of each detected object in the time series;
[0054] Combine the results of semantic segmentation and historical data to build a behavior model of the target.
[0055] Preferably, the behavior prediction formula is:
[0056]
[0057] in, t is the velocity vector of pedestrian (i) at time (t), α is the weight, prd is the predicted velocity vector based on semantic segmentation results and historical data;
[0058] Based on the results of situational reasoning, predict the target's behavior and movements in the next few seconds or frames of images; feed the prediction results back to the system to support decision-making and behavior planning.
[0059] Preferably, the step of S5 is:
[0060] Combine the semantic segmentation information with the target detection results to form a comprehensive understanding of the surrounding environment.
[0061] Combine the perceived target information and environmental perception data to understand and analyze the situation, formulate driving strategies and decisions, and plan a safe and efficient driving path based on the current vehicle position, target position, and predicted environmental changes;
[0062] During the entire process, real-time data from the perception system is continuously received and processed, and the outputs of the decision-making and control modules are updated in real time.
[0063] Compared with the prior art, the present invention has the following beneficial effects:
[0064] This application can quickly select a route by classifying target information, and can quickly select the optimal route during driving by analyzing and predicting behavior. It can also make dynamic choices based on data changes, thereby improving usage prospects and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 It is a step flow framework diagram of the present invention. DETAILED DESCRIPTION
[0066] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.
[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0068] The present invention provides a technical solution: a three-dimensional perception method of a dynamic target of an intelligent vehicle in a complex environment, the perception method comprising:
[0069] S1: Obtain data from cameras, lidar, radar and ultrasonic sensors, and fuse data from multiple sensors;
[0070] S2: Use the deep learning model of convolutional neural network for real-time target detection and classification, and combine it with the target motion model for target tracking to infer the future position and speed of the target;
[0071] S3: Converts the point cloud data acquired by the lidar into a 3D scene model, combines vision, lidar and inertial measurement unit data to build a 3D map around the vehicle in real time, and locates and tracks targets in it;
[0072] S4: Use deep learning technology to perform semantic segmentation on the images captured by the camera, divide the scene into different semantic areas, and combine the target's motion pattern, surrounding environment information and historical data to perform situational reasoning and predict the target's possible behavior and future movements;
[0073] S5: The sensed target information is transmitted to the decision-making and control system of the smart car to adjust the vehicle's driving trajectory and speed in real time.
[0074] Among them, the steps of S1 are:
[0075] Synchronize the data, obtain the timestamp, and calibrate the timestamp;
[0076] Parse the data into a uniform format;
[0077] According to application requirements and sensor capabilities, select the level of data fusion, including sensor-level fusion, feature-level fusion, and decision-level fusion;
[0078] For each sensor, the measurement accuracy and confidence are evaluated through measurement accuracy analysis;
[0079] According to the characteristics and evaluation results of the sensor, the weighted average data fusion algorithm is selected. Different weights are assigned according to the confidence of the sensor. The fused data is obtained by weighted average calculation. The weight is dynamically adjusted according to the measurement accuracy of the sensor. The specific steps are as follows:
[0080] Detailed measurement accuracy and confidence assessment for each sensor:
[0081] Analyze the measurement error range and stability of each sensor under different conditions;
[0082] Considering the reliability, working status and environmental adaptability of the sensor, the confidence assessment is carried out through the verification results and fault analysis in actual operation;
[0083] According to the evaluation results, the weight distribution strategy of each sensor in data fusion is determined, and the weight distribution is based on the measurement accuracy and confidence:
[0084] The data fusion process of performing weighted average method:
[0085] Weighted average calculation: According to the set weight, the measurement value of each sensor is weighted averaged. The calculation formula of weighted average is as follows:
[0086]
[0087] Among them, FV is the fused value, Sen i is the measurement value of the ith sensor, w i is the weight of the i-th sensor, n is the number of sensors;
[0088] The fusion result after weighted average is obtained and used for further decision-making and control of the system.
[0089] Furthermore, a data fusion process is performed to integrate the information from different sensors into a unified data representation. The specific steps include:
[0090] Convert the data from each sensor into a unified data space or coordinate system;
[0091] The final fusion data is used for system decision-making and control, providing accurate environmental perception information;
[0092] In practical applications, the accuracy and real-time performance of fused data are continuously verified, and sensor selection, weight distribution, and fusion strategies are optimized based on actual feedback.
[0093] Furthermore, the steps of S2 are:
[0094] Get real-time video stream and image sequence as input data, determine the target category and category label, use CNN for target detection and classification, and the CNN classification formula is:
[0095]
[0096] Among them, I i+m,j+n,l· is the element of the input data, W m,n,l,k is the parameter of the kth filter, b k It is k The bias of the filter, σ is the activation function, O ijk is the output of the convolution operation;
[0097] Select the SSD target detection task model;
[0098] Preprocess the input image, including normalization and resizing;
[0099] Extract features from images through convolution and pooling operations;
[0100] Identify and locate objects of interest in images and classify them into predefined categories;
[0101] Output detection box, target category and confidence score;
[0102] Target tracking is performed in conjunction with the target motion model, using the target position information detected in the previous steps as the starting point. The goal of target tracking is to maintain the target's identity between consecutive frames and predict its future position and speed.
[0103] Use recurrent neural network motion models;
[0104] The motion model formula is:
[0105] xt=RNN(xt-1,u t )]
[0106] Among them, RNN represents the transfer function of the recurrent neural network, which receives the state xt-1 at the previous moment and the input u at the current moment t , and then output the current state xt;
[0107] Initialize the tracker using the detection results in the first frame, including the target location and category;
[0108] Predict the target's future position and velocity based on its current position and motion model;
[0109] The position prediction formula is:
[0110] [xt+k=RNN(xt+k-1,ut+k-1)]
[0111] The initial conditions are the position xt and input ut at the current time t;
[0112] The speed prediction formula is:
[0113] [vt = RNN v (v t-1 ,ut)]
[0114] Among them, RNN v It is a RNN transfer function specifically used to predict speed;
[0115] Associate the detection results in the current frame with the tracked target in the previous frame;
[0116] Updates the status of the tracker, including position and velocity.
[0117] Furthermore, the steps of S3 are:
[0118] LiDAR acquires point cloud data by emitting laser beams and measuring their reflections. Visual sensors provide image data for object detection, feature extraction, and visual positioning. IMU provides the vehicle's accelerometer and gyroscope data. The point cloud data acquired by LiDAR is denoised, filtered, and segmented. Features such as surface normals, planes, edges, etc. are extracted from the point cloud data for subsequent map construction and target detection. LiDAR and visual data are used to initialize the map around the vehicle. Real-time LiDAR data is used to compare and fuse new point cloud data with existing maps.
[0119] Furthermore, vision and point cloud data are used for target detection and recognition, data from lidar, vision and IMU are fused, and a multi-target tracking algorithm is used to track the detected targets. The map is continuously updated to ensure that real-time environmental changes such as moving obstacles or new road structures are reflected. The map, vehicle status and estimated target position information are fed back to the vehicle's navigation system or driver assistance system for path planning, obstacle avoidance and navigation decisions.
[0120] Furthermore, the steps of S4 are:
[0121] Use deep learning technology to perform semantic segmentation on the image, assign each pixel in the image to a different semantic category, use the object detection model in deep learning to identify and locate the position and bounding box of each object in the semantically segmented image, and track the movement trajectory of each detected object in the time series;
[0122] Combine the results of semantic segmentation and historical data to build a behavior model of the target.
[0123] Furthermore, the behavior prediction formula is:
[0124]
[0125] in, t is the velocity vector of pedestrian (i) at time (t), α is the weight, prd is the predicted velocity vector based on semantic segmentation results and historical data;
[0126] Based on the results of situational reasoning, predict the target's behavior and movements in the next few seconds or frames of images; feed the prediction results back to the system to support decision-making and behavior planning.
[0127] Furthermore, the steps of S5 are:
[0128] Combine the semantic segmentation information with the target detection results to form a comprehensive understanding of the surrounding environment.
[0129] Combine the perceived target information and environmental perception data to understand and analyze the situation, formulate driving strategies and decisions, and plan a safe and efficient driving path based on the current vehicle position, target position, and predicted environmental changes;
[0130] During the entire process, real-time data from the perception system is continuously received and processed, and the outputs of the decision-making and control modules are updated in real time.
[0131] The above description only describes the specific implementation mode of the present invention. Various examples do not limit the essential content of the present invention. After reading the description, ordinary technicians in the relevant technical field can modify or deform the specific implementation mode described above without departing from the essence and scope of the invention.
Claims
1. A three-dimensional perception method for dynamic targets of intelligent vehicles in complex environments, characterized in that: Perception methods include: S1: Obtain data from cameras, lidar, radar and ultrasonic sensors, and fuse data from multiple sensors; S2: Use the deep learning model of convolutional neural network for real-time target detection and classification, and combine it with the target motion model for target tracking to infer the future position and speed of the target; S3: Converts the point cloud data acquired by the lidar into a 3D scene model, combines vision, lidar and inertial measurement unit data to build a 3D map around the vehicle in real time, and locates and tracks targets in it; S4: Use deep learning technology to perform semantic segmentation on the images captured by the camera, divide the scene into different semantic areas, and combine the target's motion pattern, surrounding environment information and historical data to perform situational reasoning and predict the target's possible behavior and future movements; S5: The sensed target information is transmitted to the decision-making and control system of the smart car to adjust the vehicle's driving trajectory and speed in real time.
2. The method for three-dimensional perception of dynamic targets of intelligent vehicles in complex environments according to claim 1 is characterized in that: The steps of S1 are: Synchronize the data, obtain the timestamp, and calibrate the timestamp; Parse the data into a uniform format; For each sensor, the measurement accuracy and confidence are evaluated through measurement accuracy analysis; Perform detailed measurement accuracy and confidence assessment for each sensor; Analyze the measurement error range and stability of each sensor under different conditions; According to the evaluation results, determine the weight allocation strategy of each sensor in data fusion; According to the set weight, the measurement value of each sensor is weighted averaged. The calculation formula of weighted average is as follows: Among them, FV is the fused value, Sen i is the measurement value of the ith sensor, w i is the weight of the i-th sensor, n is the number of sensors; The fusion result after weighted average is obtained and used for further decision-making and control of the system.
3. The method for three-dimensional perception of dynamic targets of intelligent vehicles in complex environments according to claim 2 is characterized by: Perform a data fusion process to combine information from different sensors into a unified data representation. The specific steps include: Convert the data from each sensor into a unified data space or coordinate system; The final fusion data is used for system decision-making and control, providing accurate environmental perception information; Continuously verify the accuracy and real-time performance of fused data, and optimize sensor selection, weight distribution, and fusion strategy based on actual feedback.
4. The method for three-dimensional perception of dynamic targets of intelligent vehicles in complex environments according to claim 1 is characterized in that: The steps of S2 are: Get real-time video stream and image sequence as input data, determine the target category and category label, use CNN for target detection and classification, and the CNN classification formula is: Among them, I i+m,j+n,l· is the element of the input data, W m,n,l,k is the parameter of the kth filter, b k It is k The bias of the filter, σ is the activation function, O ijk is the output of the convolution operation; Select the SSD target detection task model; Preprocess the input image, including normalization and resizing; Extract features from images through convolution and pooling operations; Identify and locate objects in images and classify them into predefined categories; Output detection box, target category and confidence score; Combined with the target motion model for target tracking, use the target position information detected in the previous steps as the starting point and predict its future position and velocity.
5. The method for three-dimensional perception of dynamic targets of intelligent vehicles in complex environments according to claim 4 is characterized in that: Use recurrent neural network motion models; The motion model formula is: [xt=RNN(xt-1,u t )] Among them, RNN represents the transfer function of the recurrent neural network, which receives the state xt-1 at the previous moment and the input ut at the current moment, and then outputs the state xt at the current moment; Initialize the tracker using the detection results in the first frame, including the target location and category; Predict the target's future position and velocity based on its current position and motion model; The position prediction formula is: [xt+k=RNN(xt+k-1,ut+k-1)] The initial conditions are the position xt and input ut at the current time t; The speed prediction formula is: [vt=RNN v (in t-1 ,ut)] Among them, RNN v It is the RNN transfer function specifically used to predict speed; Associate the detection results in the current frame with the tracked target in the previous frame; Updates the status of the tracker, including position and velocity.
6. The method for three-dimensional perception of dynamic targets of intelligent vehicles in complex environments according to claim 1 is characterized in that: The steps of S3 are: LiDAR acquires point cloud data by emitting laser beams and measuring their reflections. Visual sensors provide image data for object detection, feature extraction, and visual positioning. IMU provides the vehicle's accelerometer and gyroscope data. The point cloud data acquired by LiDAR is denoised, filtered, and segmented. Features such as surface normals, planes, edges, etc. are extracted from the point cloud data for subsequent map construction and target detection. LiDAR and visual data are used to initialize the map around the vehicle. Real-time LiDAR data is used to compare and fuse new point cloud data with existing maps.
7. The method for three-dimensional perception of dynamic targets of intelligent vehicles in complex environments according to claim 6 is characterized by: Use vision and point cloud data for target detection and recognition, fuse data from LiDAR, vision and IMU, use multi-target tracking algorithm to track detected targets, continuously update maps to ensure that real-time environmental changes are reflected, and feed back map, vehicle status and target position estimation information to the vehicle's navigation system or driver assistance system.
8. The method for three-dimensional perception of dynamic targets of intelligent vehicles in complex environments according to claim 1 is characterized in that: The steps of S4 are: Use deep learning technology to perform semantic segmentation on the image, assign each pixel in the image to a different semantic category, use the object detection model in deep learning to identify and locate the position and bounding box of each object in the semantically segmented image, and track the movement trajectory of each detected object in the time series; Combine the results of semantic segmentation and historical data to build a behavior model of the target.
9. The method for three-dimensional perception of dynamic targets of intelligent vehicles in complex environments according to claim 1, characterized in that: The behavior prediction formula is: in, t is the velocity vector of pedestrian (i) at time (t), α is the weight, prd is the predicted velocity vector based on semantic segmentation results and historical data; Based on the results of situational reasoning, predict the target's behavior and movement in the next few seconds or frames of images; Feed predictions back into the system to support decision making and action planning.
10. The method for three-dimensional perception of dynamic targets of intelligent vehicles in complex environments according to claim 1, characterized in that: The steps of S5 are: Combine the semantic segmentation information with the results of target detection, and combine the perceived target information and environmental perception data to perform situational understanding and analysis, formulate driving strategies and decisions, plan safe and efficient driving paths based on the current vehicle position, target position and predicted environmental changes, continuously receive and process real-time data from the perception system, and update the outputs of the decision and control modules in real time.
Citation Information
Cited By
Self-adaptive positioning method and system for intelligent factory
CN119494947A
Road three-dimensional model construction method and device, computer equipment and storage medium
CN119741436A
Road three-dimensional model construction method, device, computer equipment and storage medium
CN119741436B
Intelligent agent and urban road interaction real-time scene analysis method
CN120340262A
A real-time scene analysis method for the interaction between intelligent agents and urban roads
CN120340262B
