Multi-dimensional perception automatic driving avoidance method and device
Through multi-dimensional perception and deep learning technology, combined with DQN algorithm, we can identify obstacle types and generate safety avoidance solutions, which solves the problem of insufficient detection accuracy and robustness of obstacles in autonomous driving systems and improves driving safety.
Patent Information
- Application Number
- CN202410983475.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-22
AI Technical Summary
The existing autonomous driving system has low detection accuracy in obstacle detection, is greatly affected by ambient light, and is insufficient real-time and robust.
Using a multi-dimensional perception method, real-time perception data of the agent during driving, including the agent's driving data, road visual information, thermal energy distribution data and three-dimensional point cloud data, combined with deep neural network and DQN algorithm, real-time driving information and Avoidance solutions, including obstacle type identification, distance calculation and path planning.
It improves the accuracy and robustness of obstacle detection, realizes real-time monitoring and reporting of obstacles on the road, provides safe avoidance routes, and improves driving safety.
Smart Images

Figure CN118928462A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of path planning, and in particular to a multi-dimensional perception autonomous driving avoidance method and device. Background Art
[0002] Path planning algorithms are a key technology for intelligent agents to avoid collisions when interacting with their environment. Based on the requirements of their work environment, the agent uses various optimization metrics to find the optimal path from a starting node to a destination node, such as minimizing path length, energy consumption, or computation time.
[0003] Complex environment perception technology is a key part of path planning. It collects data through sensors and, after corresponding processing and analysis of the data, can realize functions such as traffic road detection, positioning and motion estimation, vehicle identification and tracking, obstacle detection, traffic light recognition, and traffic sign recognition.
[0004] However, the current obstacle detection through environmental perception has low detection accuracy and is greatly affected by ambient lighting. It has certain limitations in terms of real-time performance, accuracy, and robustness. Summary of the Invention
[0005] In view of the above problems, the present application is proposed to provide a multi-dimensional perception autonomous driving avoidance method and device that overcomes the above problems or at least partially solves the above problems, including:
[0006] A multi-dimensional perception autonomous driving avoidance method includes the following steps:
[0007] Acquiring real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road, and three-dimensional point cloud data of the environment;
[0008] Determining corresponding real-time obstacle features on the driving road based on the real-time perception data;
[0009] Determining a corresponding obstacle type based on the real-time obstacle characteristics;
[0010] Generating real-time driving information based on the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the centerline of the lane;
[0011] A real-time avoidance plan is generated according to the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.
[0012] Furthermore, the step of determining the corresponding obstacle type based on the real-time obstacle feature includes:
[0013] fusing the real-time obstacle features;
[0014] The corresponding obstacle type is determined according to the fused obstacle features.
[0015] Furthermore, the step of generating real-time driving information based on the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane, includes:
[0016] Generating a distance between the agent and the obstacle based on the real-time obstacle feature and the real-time perception data;
[0017] determining a lane centerline based on the real-time perception data;
[0018] A distance between the agent and the lane centerline is generated based on the lane centerline and the real-time perception data.
[0019] Furthermore, the step of determining the lane centerline based on the real-time perception data includes:
[0020] performing morphological processing on the visual information;
[0021] Segmenting the visual information and determining corresponding weights, and determining a minimum bounding rectangle for each portion of the visual information;
[0022] The center points of the lines of the minimum circumscribed rectangle of each part are weighted and summed to obtain the lane center line.
[0023] Furthermore, the step of generating a real-time avoidance plan based on the obstacle type and the real-time driving information includes:
[0024] Based on the DQN algorithm, the Q value of the driving path is estimated according to the obstacle type and the real-time driving information; wherein the DQN algorithm estimates the Q value of each possible action through a deep neural network, the deep neural network includes a main network and a target network, and the Q value includes a main Q value output by the main network and a target Q value of the path planned by the target network;
[0025] The real-time avoidance solution is generated according to the main Q value and the target Q value.
[0026] Furthermore, the step of estimating the Q value of the driving path based on the obstacle type and the real-time driving information includes:
[0027] generating classified safety data based on the obstacle type and the distance between the agent and the obstacle;
[0028] Inputting the classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the centerline of the lane into the main network, and outputting the main Q value;
[0029] The classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane are input into the target network, and the target Q value is output.
[0030] Furthermore, the calculation formula of the main Q value is as follows:
[0031] Q(s,a) def =Q(s,a)+α*(R+γmax(Q(s′,a′))-Q(s,a))
[0032] Where s is the state, a is the action, s′ is the next state, a′ is the optimal action, α is the learning rate, r is the reward, and γ is the discount factor;
[0033] The target Q value is calculated as follows:
[0034] TargetQ(s,a) def =R+γ*max(Q_target(s′,a′))
[0035] Among them, Q_targrt is the Q value of the target network.
[0036] A multi-dimensional perception autonomous driving avoidance device, comprising:
[0037] A data acquisition module is used to obtain real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road, and three-dimensional point cloud data of the environment;
[0038] A feature extraction module is used to determine corresponding real-time obstacle features on the driving road based on the real-time perception data;
[0039] An obstacle classification module, configured to determine a corresponding obstacle type based on the real-time obstacle features;
[0040] a data processing module, configured to generate real-time driving information based on the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes the distance between the agent and the obstacle and the distance between the agent and the centerline of the lane;
[0041] A path planning module is used to generate a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.
[0042] A device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the steps of the multi-dimensional perception autonomous driving avoidance method as described above are implemented.
[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-dimensional perception autonomous driving avoidance method as described above.
[0044] This application has the following advantages:
[0045] In the embodiments of the present application, relative to the problems in the prior art of "low detection accuracy, great influence by ambient light, and certain limitations in real-time, accuracy and robustness", the present application provides a solution to a multi-dimensional perception autonomous driving avoidance method, specifically: obtaining real-time perception data of an intelligent agent during driving; wherein, the real-time perception data includes intelligent agent driving data, road visual information, thermal energy distribution data on the road and three-dimensional point cloud data of the environment; determining corresponding real-time obstacle features on the driving road based on the real-time perception data; determining corresponding obstacle types based on the real-time obstacle features; generating real-time driving information based on the real-time obstacle features and the real-time perception data; wherein, the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane; generating a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein, the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking. The present invention improves the accuracy and robustness of obstacle detection by comprehensively sensing obstacles on the road and comprehensively analyzing the perception data from different sensors. It also analyzes the data in real time to achieve real-time monitoring and reporting of obstacles on the road, and can provide safe avoidance routes based on the detected obstacle information, thereby improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0047] Figure 1 This is a flowchart of the steps of a multi-dimensional perception autonomous driving avoidance method provided by an embodiment of the present application;
[0048] Figure 2 This is a flowchart of a multi-dimensional perception autonomous driving avoidance method provided by an embodiment of the present application;
[0049] Figure 3 This is a structural block diagram of a multi-dimensional perception autonomous driving avoidance device provided in one embodiment of the present application;
[0050] Figure 4 It is a structural diagram of a computer device provided by one embodiment of the present invention. DETAILED DESCRIPTION
[0051] To make the objectives, features, and advantages of this application more readily apparent, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It is apparent that the embodiments described are only a portion of the embodiments of this application, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments in this application without inventive effort are also within the scope of protection of this application.
[0052] Reference Figure 1 and Figure 2 , showing a multi-dimensional perception autonomous driving avoidance method provided by an embodiment of the present application;
[0053] The method comprises:
[0054] S110, acquiring real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road, and three-dimensional point cloud data of the environment;
[0055] S120, determining corresponding real-time obstacle features on the driving road based on the real-time perception data;
[0056] S130. Determine a corresponding obstacle type according to the real-time obstacle feature;
[0057] S140: Generate real-time driving information based on the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes a distance between the agent and the obstacle and a distance between the agent and a lane centerline;
[0058] S150. Generate a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing, and parking.
[0059] In the embodiments of the present application, relative to the problems in the prior art of "low detection accuracy, great influence by ambient light, and certain limitations in real-time, accuracy and robustness", the present application provides a solution to a multi-dimensional perception autonomous driving avoidance method, specifically: obtaining real-time perception data of an intelligent agent during driving; wherein, the real-time perception data includes intelligent agent driving data, road visual information, thermal energy distribution data on the road and three-dimensional point cloud data of the environment; determining corresponding real-time obstacle features on the driving road based on the real-time perception data; determining corresponding obstacle types based on the real-time obstacle features; generating real-time driving information based on the real-time obstacle features and the real-time perception data; wherein, the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane; generating a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein, the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking. The present invention improves the accuracy and robustness of obstacle detection by comprehensively sensing obstacles on the road and comprehensively analyzing the perception data from different sensors. It also analyzes the data in real time to achieve real-time monitoring and reporting of obstacles on the road, and can provide safe avoidance routes based on the detected obstacle information, thereby improving driving safety.
[0060] Below, a multi-dimensional perception autonomous driving avoidance method in this exemplary embodiment will be further explained.
[0061] As described in step S110, real-time perception data of the intelligent agent during driving is obtained; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road and three-dimensional point cloud data of the environment.
[0062] It should be noted that this embodiment uses multiple hardware to obtain multi-dimensional real-time perception data. These sensors are each responsible for sensing different physical information, such as image, distance, speed, etc. By comprehensively utilizing the data from these sensors, more comprehensive and accurate road obstacle information can be obtained.
[0063] The hardware used includes: infrared thermal imaging module, lidar module and multiple sensors.
[0064] Infrared thermal imaging module: This module consists of an infrared thermal imager, optical lens, image sensor, signal processor, and interface circuitry. It uses infrared radiation emitted by an object to generate a thermal image. By measuring and recording the surface temperature of different areas of the object, it can display the object's thermal distribution in real time. It operates in various environments and lighting conditions and is unaffected by visible light. Using infrared thermal imaging technology, it can sense and capture the distribution of heat energy on the road. Using an infrared camera or thermal imaging sensor, the module acquires real-time heat energy information on the road and converts it into a digital signal for subsequent processing.
[0065] Lidar module: The Lidar module provides high-precision 3D perception data for obstacle detection and identification. It uses laser beams to scan the surrounding environment, measuring and recording information such as the distance, shape, and position of obstacles for tasks such as obstacle detection, mapping, and target tracking. The Lidar module emits laser beams and receives reflected signals to obtain 3D point cloud data of the road environment. This point cloud data is then processed through filtering, clustering, and feature extraction to identify, classify, and track obstacles on the road. Finally, based on the detection results, it generates corresponding alarm signals or control commands for applications such as driver assistance and traffic safety.
[0066] Sensors include cameras, radars, ultrasonic sensors, and inertial measurement units (IMUs).
[0067] Cameras: These capture images of the road and surroundings. Through image processing and computer vision algorithms, they can detect obstacles such as road signs, lane information, traffic lights, pedestrians, and vehicles, providing visual perception capabilities.
[0068] Radar: Radar can transmit radio waves and receive their reflected signals to measure the range, speed and direction of the target.
[0069] Ultrasonic sensor: Sends ultrasonic pulses and measures their return time to determine distance. It is usually used for close-range obstacle detection.
[0070] Inertial Measurement Unit (IMU): Consists of an accelerometer and a gyroscope, it measures the acceleration and angular velocity of a device. Through integration, it can obtain information such as the device's attitude, position, and motion state.
[0071] The infrared thermal imaging module is used to obtain infrared images of the road scene, and the lidar module is used to obtain three-dimensional point cloud data of the road scene. Sensors include but are not limited to high-definition cameras, millimeter-wave radars, lidars, and ultrasonic sensors. High-definition cameras can capture visual information of the road environment for subsequent image processing and obstacle identification. Millimeter-wave radars and lidars can provide accurate data on the distance, speed, and direction of obstacles. Ultrasonic sensors can achieve high-precision obstacle detection within a short distance. The present invention utilizes multiple sensors and multi-dimensional perception data for comprehensive analysis to achieve accurate, real-time, and comprehensive detection of obstacles on the road, providing drivers and traffic management departments with reliable road safety information.
[0072] As an example, multiple hardware devices are used to acquire multi-dimensional perception data, and navigation map data information, including road networks, traffic signs, traffic lights, and other information, can also be obtained from a navigation map database. The user can also set the starting and target locations through the user interface or other means. After receiving this information, the autonomous driving avoidance device of the present invention uses it as input to calculate the avoidance plan.
[0073] In a specific implementation, the present application also has a data storage and management module, which can persistently store the multi-dimensional perception data collected by the sensors to ensure the integrity and availability of the data. The module provides efficient data indexing and query functions, allowing researchers to easily retrieve and obtain data for a specific time period, a specific road section or a specific type of obstacle. The module supports the addition, deletion and modification of data to facilitate data cleaning and maintenance. At the same time, the module can also back up and restore data to ensure data security and reliability. The module provides data analysis tools and algorithm interfaces for research work such as data mining, trend analysis and pattern recognition. Researchers can use this module to conduct in-depth analysis of data collected over a long period of time to discover potential patterns and trends. The module supports data visualization functions and can display data in the form of charts, curves, etc., allowing researchers to understand and analyze data more intuitively.
[0074] As described in step S120, the corresponding real-time obstacle features on the driving road are determined based on the real-time perception data.
[0075] As an example, a real-time data processing module receives, processes, and analyzes sensor-generated data. This module can receive sensor data from a variety of sensors, including cameras, radars, lidars, and ultrasonic sensors. This data includes multi-dimensional information such as images, distance, and speed. The real-time data processing module preprocesses the sensor data from these sensors, including noise removal, filtering, and data smoothing. This allows for real-time monitoring and reporting of road obstacles, improving data quality and accuracy. The collected raw data is preprocessed to remove noise and interference, improving data quality. In this step, the road obstacle detection device preprocesses the collected raw data through filtering, Gaussian blurring, denoising, correction, and image conversion to Lab color space to remove noise, filter out interference, and improve data quality and accuracy. For each sensor's real-time sensor data, a corresponding feature representation needs to be extracted. This can include image features, distance features, speed features, and more. The goal of feature extraction is to extract information from the raw data that can describe the properties of the obstacle. Deep learning can be used to extract feature information of obstacles from preprocessed data. This feature information includes the object's location, shape, size, texture, etc. By extracting this feature information, the ability to identify obstacles can be further enhanced.
[0076] As described in step S130, the corresponding obstacle type is determined according to the real-time obstacle feature.
[0077] As an example, after obtaining the feature representation of each sensor, in order to improve the accuracy of obstacle detection and classification, the perception data of different sensors needs to be fused. Since different sensors may have different sampling rates, coordinate systems and data formats, the perception data from different sensors need to be aligned first. This embodiment is implemented through technologies such as timestamp matching and coordinate conversion, so that the data can be compared and fused at the same time and coordinate system. Before data fusion, the perception data from different sensors are preprocessed, and the corresponding feature representation is extracted for the perception data of each sensor. After obtaining the feature representation of each sensor, the data fusion algorithm will fuse these features. Fusion can be performed in a variety of ways, such as weighted averaging, decision cascade, probabilistic fusion, etc.
[0078] The real-time data processing module uses pre-set algorithms and thresholds to detect obstacles on the road in real time. Based on features and patterns in sensor data, it determines the presence of obstacles, locates them, and tracks them. When an obstacle is detected, the real-time data processing module generates real-time reports and alerts. These reports and alerts promptly notify drivers and traffic management authorities so they can take appropriate action. When an obstacle is detected, the obstacle classification module classifies it and determines its corresponding obstacle type. Specifically, using machine learning and computer vision algorithms, a classifier classifies and identifies detected obstacles based on extracted feature information, distinguishing between different types of obstacles, such as vehicles, pedestrians, and bicycles. Accurately classifying and identifying detected obstacles based on sensor data provides important information for subsequent response strategies. The classifier can be a machine learning-based model, such as a support vector machine (SVM), random forest, or convolutional neural network (CNN). Accurate obstacle classification provides important information for subsequent decision-making and actions.
[0079] As described in step S140, real-time driving information is generated based on the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane.
[0080] In one embodiment of the present invention, the specific process of "generating real-time driving information based on the real-time obstacle characteristics and the real-time perception data in step S140; wherein the real-time driving information includes the distance between the intelligent body and the obstacle and the distance between the intelligent body and the center line of the lane" can be further explained in combination with the following description.
[0081] As described in the following steps, generating the distance between the intelligent agent and the obstacle based on the real-time obstacle characteristics and the real-time perception data;
[0082] Determine a lane centerline based on the real-time perception data as described in the following steps;
[0083] As described in the following steps, a distance between the intelligent agent and the lane centerline is generated based on the lane centerline and the real-time perception data.
[0084] In one embodiment of the present invention, the specific process of "generating the distance between the intelligent agent and the obstacle based on the real-time obstacle characteristics and the real-time perception data" can be further explained in combination with the following description.
[0085] As an example, when an obstacle is detected, the real-time data processing module calculates the distance between the obstacle and the intelligent body through the perception data collected by the camera, radar or lidar, and generates real-time reports and alarms.
[0086] In one embodiment of the present invention, the specific process of “determining the lane center line based on the real-time perception data” can be further explained in combination with the following description.
[0087] Performing morphological processing on the visual information as described in the following steps;
[0088] As described in the following steps, the visual information is segmented and corresponding weights are determined to determine the minimum bounding rectangle of each portion of the visual information;
[0089] As described in the following steps, the center points of the lines of each minimum circumscribed rectangle are weighted and summed to obtain the lane center line.
[0090] Specifically, the preprocessed image is stripped of all colors except white, and morphological processing is performed to alter the shape and size of objects and remove noise. The image is segmented into three parts: upper, middle, and lower. Weights are assigned to each part. Contour detection is then used to locate the target object. Contours are sorted by area and their minimum bounding rectangles are obtained. The lane center point is then determined by summing the weighted center points of the three lines.
[0091] In one embodiment of the present invention, the specific process of "generating the distance between the intelligent agent and the lane boundary based on the lane boundary and the real-time perception data" can be further explained in combination with the following description.
[0092] As an example, when a lane boundary is detected, the real-time data processing module calculates the distance between the intelligent agent and the lane boundary through the perception data collected by the camera, radar or lidar.
[0093] In one embodiment of the present invention, the specific process of "generating the distance between the intelligent agent and the lane centerline based on the lane centerline and the real-time perception data" can be further explained in combination with the following description.
[0094] For example, when a lane centerline is detected, the real-time data processing module calculates the distance between the agent and the lane centerline using sensory data collected by cameras, radar, or lidar. Because lane boundaries can pose safety risks, calculating the distance between the agent and the lane centerline ensures that the agent maintains a certain distance from the lane boundary, allowing the agent to travel in the middle of the lane.
[0095] In one embodiment of the present invention, the specific process of "generating a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking" in step S150 can be further explained in combination with the following description.
[0096] It should be noted that the results of road condition detection are taken into consideration when planning avoidance plans. These results describe information such as the location, type, and status of obstacles on the road. Avoidance plans and navigation are provided based on the detected obstacle information, providing a safe driving route. By comprehensively considering obstacles and their dynamic changes, potential collision risks can be avoided and driving safety and efficiency can be improved.
[0097] As described in the following steps, based on the DQN algorithm, the Q value of the driving path is estimated according to the obstacle type and the real-time driving information; wherein the DQN algorithm estimates the Q value of each possible action through a deep neural network, the deep neural network includes a main network and a target network, and the Q value includes a main Q value output by the main network and a target Q value of the path planned by the target network;
[0098] As described in the following steps, the real-time avoidance solution is generated according to the main Q value and the target Q value.
[0099] In one embodiment of the present invention, the specific process of “estimating the Q value of the driving path according to the obstacle type and the real-time driving information” may be further explained in combination with the following description.
[0100] Generating classified safety data according to the obstacle type and the distance between the intelligent agent and the obstacle as described in the following steps;
[0101] As described in the following steps, the classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the centerline of the lane are input into the main network, and the main Q value is output;
[0102] As described in the following steps, the classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane are input into the target network, and the target Q value is output.
[0103] As an example, categorized safety data can be used to estimate specific safety values for different types of obstacles. The farther the agent is from the obstacle, the higher the safety value, while the closer the agent is to the obstacle, the lower the safety value. The closer the agent is to the center line of the lane, the higher the safety value, while the farther the agent is from the center line, the lower the safety value.
[0104] The classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the lane centerline are input into the target network, and the main Q value of the main network planning path is output. The main Q value of the main network planning path is calculated as follows:
[0105] Q(s,a) def =Q(s,a)+α*(R+γmax(Q(s′,a′))-Q(s,a))
[0106] Among them, s is the state, a is the action, s′ is the next state, a′ is the optimal action, α is the learning rate, r is the reward, and γ is the discount factor.
[0107] The classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the lane centerline are input into the target network, and the target Q value of the target network planning path is output. The target Q value of the target network planning path is calculated as follows:
[0108] TargetQ(s,a) def =R+γ*max(Q_target(s′,a′))
[0109] Among them, Q_targrt is the target Q value of the target network.
[0110] Then, the loss function is generated by the calculation formula of the main Q value of the main network planning path and the calculation formula of the target Q value of the target network planning path. The calculation formula of the loss function is as follows:
[0111] Loss=MSE(Q(s,a),TargetQ(s,a))
[0112] The path planning module will eventually output the calculated real-time avoidance plan. The real-time avoidance plan includes normal driving, decelerated driving, lane changing, parking, left / right deviation, etc. The real-time avoidance plan can be passed to the intelligent driving system to achieve automatic driving or provide navigation guidance to the driver. When the distance between the intelligent agent and the obstacle is less than the preset threshold, the obstacle avoidance logic is triggered, and the intelligent agent can slow down, change lanes or stop, and adjust the driving direction of the intelligent agent to avoid collision with the obstacle. When the distance between the intelligent agent and the obstacle is greater than the preset threshold, normal driving is carried out. When the distance between the intelligent agent and the obstacle is greater than the preset threshold, but deviates from the center line of the lane, the intelligent agent deviates to the left / right so that the intelligent agent can drive along the center line of the lane.
[0113] As another example, when, in addition to acquiring multidimensional perception data using multiple hardware devices, navigation map data including road networks, traffic signs, traffic lights, and other information, as well as user-set starting and destination information, is also acquired, a real-time avoidance plan can be generated based on the main Q value, the target Q value, the navigation map data, and the starting and destination locations. In this case, the real-time avoidance plan includes not only normal driving, decelerated driving, lane changing, and stopping, but also path coordinates, turn instructions, driving distance, and estimated time. This information can be transmitted to the intelligent driving system to enable autonomous driving or provide navigation guidance to the driver. For example, if an obstacle is detected ahead and the driver should slow down or stop, but a lane change is imminent, the driver can change lanes in advance without slowing down or stopping. Another example is driving in the rightmost lane. If an obstacle is detected ahead and the driver should change lanes to the left, but a right turn is imminent, the driver can slow down or stop instead of changing lanes.
[0114] In one embodiment of the present application, the real-time avoidance plan and obstacle identification results are displayed through an output unit, including but not limited to a display screen, voice broadcast, etc. The road obstacle detection device displays the results of identification and classification through an output unit. By displaying the identification results, the driver or system operator can obtain information about road obstacles in a timely manner to make corresponding decisions and actions. The present invention is also equipped with a user interface for displaying real-time road obstacle information to the driver, providing alarms and suggestions, and helping the driver make correct decisions. The user interface displays real-time road obstacle information to the driver, and the driver can view the detected obstacles through the interface and obtain relevant alarms and suggestions. The interface can display information such as the location, type and distance of the obstacle in a graphical manner, so that the driver can understand the road conditions more intuitively.
[0115] In one embodiment of the present application, the steps also include updating the perception data in real time and adaptively adjusting the perception parameters based on changes in the road environment and weather conditions. In this step, the road obstacle detection device updates the perception data in real time and adaptively adjusts the perception parameters based on changes in the road environment and weather conditions. For example, when road conditions change or weather conditions are severe, the perception parameters can be adjusted using an adaptive algorithm to ensure detection accuracy and stability. Such real-time updates and adaptive adjustments can improve the system's adaptability to varying road environments.
[0116] The advantages of the present invention include but are not limited to the following aspects:
[0117] Multi-dimensional perception: The present invention uses multiple sensors to obtain multi-dimensional perception data of the road environment, which can fully perceive obstacles on the road and improve the accuracy and reliability of detection.
[0118] Data fusion: The data fusion algorithm comprehensively analyzes the perception data from different sensors, fully utilizing the advantages of each sensor to improve the accuracy and robustness of obstacle detection.
[0119] Real-time: The present invention is equipped with a real-time data processing module that can promptly receive, process and analyze the data generated by sensors, realize real-time monitoring and reporting of obstacles on the road, and provide immediate safety alerts and suggestions.
[0120] Efficiency: The system uses an efficient obstacle classification module to accurately classify and identify detected obstacles, providing drivers and traffic management departments with detailed obstacle information and supporting targeted response strategies.
[0121] Path planning and navigation: The present invention is equipped with a path planning module, which can provide the driver with a safe avoidance plan based on the detected obstacle information, helping the driver to avoid obstacles and improve driving safety.
[0122] User-friendly interface: The present invention provides an intuitive and friendly user interface, which displays real-time road obstacle information to the driver, enabling him to clearly understand the road conditions and make corresponding decisions.
[0123] Data storage and management: The present invention is equipped with a data storage and management module, which can effectively store and manage the collected road obstacle data, providing support for long-term data analysis and trend prediction.
[0124] Wide range of application scenarios: The present invention can be applied to the fields of autonomous driving, intelligent transportation, or robot navigation, etc.
[0125] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0126] Reference Figure 3 , shows a multi-dimensional perception autonomous driving avoidance device provided by an embodiment of the present application, including a central control module for controlling the operation of each module and outputting a planning result, and a data acquisition module, an image processing module, a data fusion module, a real-time data processing module, an obstacle classification module, and a path planning module respectively communicatively connected to the central control module; the image processing module is used to process and analyze image data, and the central control module is used to control the operation of each module and output an obstacle detection result;
[0127] The data acquisition module 310 is used to obtain real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road, and three-dimensional point cloud data of the environment;
[0128] A feature extraction module 320 is used to determine corresponding real-time obstacle features on the driving road based on the real-time perception data;
[0129] The obstacle classification module 330 is configured to determine the corresponding obstacle type based on the real-time obstacle features;
[0130] a data processing module 340 for generating real-time driving information based on the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes the distance between the agent and the obstacle and the distance between the agent and the centerline of the lane;
[0131] The path planning module 350 is used to generate a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.
[0132] In one embodiment of the present invention, the obstacle classification module 330 includes:
[0133] A fusion submodule, configured to fuse the real-time obstacle features;
[0134] The classification submodule is used to determine the corresponding obstacle type based on the fused obstacle features.
[0135] In one embodiment of the present invention, the data processing module 340 includes:
[0136] The obstacle distance calculation submodule is used to generate the distance between the intelligent agent and the obstacle based on the real-time obstacle characteristics and the real-time perception data;
[0137] a lane centerline calculation submodule, configured to determine the lane centerline based on the real-time sensing data;
[0138] The lane centerline distance calculation submodule is used to generate the distance between the intelligent agent and the lane centerline based on the lane centerline and the real-time perception data.
[0139] In one embodiment of the present invention, the lane centerline calculation submodule includes:
[0140] a pre-processing unit, configured to perform morphological processing on the visual information;
[0141] A weighting unit, configured to segment the visual information and determine corresponding weights, thereby determining a minimum bounding rectangle for each portion of the visual information;
[0142] The summing unit is used to perform weighted summation on the center points of the lines of each minimum circumscribed rectangle to obtain the lane center line.
[0143] In one embodiment of the present invention, the path planning module 350 includes:
[0144] A Q-value calculation submodule is configured to estimate the Q-value of the driving path based on the obstacle type and the real-time driving information using a DQN algorithm; wherein the DQN algorithm estimates the Q-value of each possible action using a deep neural network, the deep neural network including a main network and a target network, and the Q-value includes a main Q-value output by the main network and a target Q-value of the path planned by the target network;
[0145] The solution generation submodule is used to generate the real-time avoidance solution according to the main Q value and the target Q value.
[0146] In one embodiment of the present invention, the Q value calculation submodule includes:
[0147] a classification safety unit, configured to generate classification safety data based on the obstacle type and the distance between the agent and the obstacle;
[0148] a main Q-value calculation unit, configured to input the classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the lane centerline into the main network, and output the main Q-value;
[0149] The target Q value calculation unit is used to input the classified safety data, the distance between the intelligent agent and the obstacle, and the distance between the intelligent agent and the center line of the lane into the target network, and output the target Q value.
[0150] In one embodiment of the present invention,
[0151] The main Q value calculation unit includes:
[0152] Q(s,a) def =Q(s,a)+α*(R+γmax(Q(s′,a′))-Q(s,a))
[0153] Where s is the state, a is the action, s′ is the next state, a′ is the optimal action, α is the learning rate, r is the reward, and γ is the discount factor;
[0154] The target Q value calculation unit includes:
[0155] TargetQ(s,a) def =R+γ*max(Q_target(s′,a′))
[0156] Among them, Q_targrt is the Q value of the target network.
[0157] Reference Figure 4 , showing a computer device of a multi-dimensional perception automatic driving avoidance method of the present invention, which may specifically include the following:
[0158] The computer device 12 is a general-purpose computing device. The components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).
[0159] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor, or a local bus 18 that utilizes any of a variety of bus 18 architectures. Examples of such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.
[0160] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.
[0161] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 configured to perform the functions of various embodiments of the present invention.
[0162] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in a memory. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which, or some combination thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0163] The computer device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, a camera, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN)), a wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34.
[0164] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a multi-dimensional perception automatic driving avoidance method provided in an embodiment of the present invention.
[0165] That is, when the processing unit 16 executes the program, it realizes: obtaining the real-time perception data of the intelligent agent during driving; wherein, the real-time perception data includes the intelligent agent driving data, road visual information, thermal energy distribution data on the road and three-dimensional point cloud data of the environment; determining the corresponding real-time obstacle features on the driving road based on the real-time perception data; determining the corresponding obstacle type based on the real-time obstacle features; generating real-time driving information based on the real-time obstacle features and the real-time perception data; wherein, the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane; generating a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein, the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.
[0166] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the multi-dimensional perception autonomous driving avoidance method provided in all embodiments of the present application is implemented:
[0167] That is, when the program is executed by the processor, it is implemented as follows: obtaining real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road and three-dimensional point cloud data of the environment; determining corresponding real-time obstacle features on the driving road based on the real-time perception data; determining the corresponding obstacle type based on the real-time obstacle features; generating real-time driving information based on the real-time obstacle features and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane; generating a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.
[0168] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0169] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0170] The computer program code for performing the operations of the present invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.
[0171] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0172] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.
[0173] The above is a detailed introduction to the multi-dimensional perception autonomous driving avoidance method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A multi-dimensional perception autonomous driving avoidance method, characterized in that: Includes steps: Acquire real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road and three-dimensional point cloud data of the environment; Determining corresponding real-time obstacle features on the driving road according to the real-time perception data; Determining a corresponding obstacle type according to the real-time obstacle feature; Generating real-time driving information according to the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane; A real-time avoidance plan is generated according to the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.
2. The method according to claim 1, characterized in that The step of determining the corresponding obstacle type according to the real-time obstacle feature comprises: fusing the real-time obstacle features; The corresponding obstacle type is determined according to the fused obstacle features.
3. The method according to claim 1, characterized in that The step of generating real-time driving information based on the real-time obstacle features and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane, includes: Generating a distance between the agent and the obstacle according to the real-time obstacle feature and the real-time perception data; Determining a lane centerline based on the real-time perception data; The distance between the agent and the center line of the lane is generated according to the center line of the lane and the real-time perception data.
4. The method according to claim 3, characterized in that The step of determining the lane centerline based on the real-time perception data includes: performing morphological processing on the visual information; Segmenting the visual information and determining corresponding weights, and determining a minimum bounding rectangle for each portion of the visual information; The center points of the lines of each part's minimum circumscribed rectangle are weighted and summed to obtain the lane center line.
5. The method according to claim 1, characterized in that The step of generating a real-time avoidance solution according to the obstacle type and the real-time driving information comprises: Based on the DQN algorithm, the Q value of the driving path is estimated according to the obstacle type and the real-time driving information; wherein the DQN algorithm estimates the Q value of each possible action through a deep neural network, the deep neural network includes a main network and a target network, and the Q value includes a main Q value output by the main network and a target Q value of the path planned by the target network; The real-time avoidance solution is generated according to the main Q value and the target Q value.
6. The method according to claim 5, characterized in that The step of estimating the Q value of the driving path according to the obstacle type and the real-time driving information comprises: generating classified safety data according to the obstacle type and the distance between the agent and the obstacle; Input the classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane into the main network, and output the main Q value; The classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane are input into the target network, and the target Q value is output.
7. The method according to claim 6, characterized in that The calculation formula of the main Q value is as follows: Q(s,a) def =Q(s,a)+α*(R+γmax(Q(s′,a′))-Q(s,a)) Among them, s is the state, a is the action, s′ is the next state, a′ is the optimal action, α is the learning rate, r is the reward, and γ is the discount factor; The calculation formula of the target Q value is as follows: TargetQ(s,a)def=R+γ*max(Q target(s′,a′)) Among them, Q_targrt is the target Q value.
8. A multi-dimensional perception automatic driving avoidance device, characterized in that: include: A data acquisition module is used to obtain real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road and three-dimensional point cloud data of the environment; A feature extraction module, used to determine corresponding real-time obstacle features on the driving road according to the real-time perception data; An obstacle classification module, used to determine the corresponding obstacle type according to the real-time obstacle characteristics; A data processing module, configured to generate real-time driving information according to the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes a distance between the intelligent agent and the obstacle and a distance between the intelligent agent and the center line of the lane; A path planning module is used to generate a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.
9. A device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Intelligent device, control method and device thereof
CN109109863A
Method for automatic lane changing decision making of vehicle, vehicle-mounted device and storage medium
CN110562258A
Automatic driving lane changing decision and control method based on DQN series reinforcement learning algorithm
CN115158349A
Vehicle forward collision early warning method and system based on real-time response time of driver
CN116331246A
Target tracking method and system based on vehicle end and road end sensing fusion, and medium
CN118097614A
Cited By
Control method and system for realizing anti-collision port machine of L4-level container truck and medium
CN119428652A
Parking alarm system and method and vehicle
CN119749585A
