A multi-dimensional perception automatic driving avoidance method and device

Abstract

The present application provides a multi-dimensional perception automatic driving avoidance method and device, the method comprising: obtaining real-time perception data of an intelligent body during driving; the real-time perception data includes intelligent body driving data, road visual information, heat energy distribution data on the road and three-dimensional point cloud data; determining real-time obstacle characteristics on the driving road based on real-time perception data; determining obstacle types based on real-time obstacle characteristics; generating real-time driving information based on real-time obstacle characteristics and real-time perception data; wherein the real-time driving information includes the distance between the intelligent body and the obstacle and the distance between the intelligent body and the center line of the lane; generating a real-time avoidance plan based on the obstacle type and real-time driving information. By fully perceiving obstacles on the road, the accuracy and robustness of obstacle detection are improved; and by analyzing the data in real time, a safe avoidance route can be provided based on the detected obstacle information, thereby improving driving safety.

Description

Technical Field

[0001] The present application relates to the field of path planning, and in particular to a multi-dimensional perception automatic driving avoidance method and device. Background Art

[0002] Path planning algorithm is one of the key technologies for intelligent agents to ensure that they do not collide when interacting with the environment. According to the requirements of their working environment, the intelligent agent finds the best path from the starting node to the target node through different optimization indicators, such as the shortest path length, the minimum energy consumption or the shortest path calculation time.

[0003] Complex environment perception technology is a key part of path planning. It collects data through sensors and, after corresponding processing and analysis of the data, can realize functions such as traffic road detection, positioning and motion estimation, vehicle identification and tracking, obstacle detection, traffic light recognition, and traffic sign recognition.

[0004] However, the current obstacle detection through environmental perception has low detection accuracy and is greatly affected by ambient lighting, and has certain limitations in terms of real-time performance, accuracy, and robustness. Summary of the invention

[0005] In view of the above problems, the present application is proposed to provide a multi-dimensional perception automatic driving avoidance method and device that overcomes the above problems or at least partially solves the above problems, including:

[0006] A multi-dimensional perception automatic driving avoidance method, comprising the steps of:

[0007] Acquire real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road and three-dimensional point cloud data of the environment;

[0008] Determining corresponding real-time obstacle features on the driving road according to the real-time perception data;

[0009] Determining a corresponding obstacle type according to the real-time obstacle feature;

[0010] Generating real-time driving information according to the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane;

[0011] A real-time avoidance plan is generated according to the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.

[0012] Furthermore, the step of determining the corresponding obstacle type according to the real-time obstacle feature includes:

[0013] fusing the real-time obstacle features;

[0014] The corresponding obstacle type is determined according to the fused obstacle features.

[0015] Further, the step of generating real-time driving information based on the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane, includes:

[0016] Generating a distance between the agent and the obstacle according to the real-time obstacle feature and the real-time perception data;

[0017] Determining a lane centerline based on the real-time perception data;

[0018] The distance between the agent and the center line of the lane is generated according to the center line of the lane and the real-time perception data.

[0019] Furthermore, the step of determining the lane centerline based on the real-time perception data includes:

[0020] performing morphological processing on the visual information;

[0021] Segmenting the visual information and determining corresponding weights, and determining a minimum bounding rectangle for each portion of the visual information;

[0022] The center points of the lines of each part's minimum circumscribed rectangle are weighted and summed to obtain the lane center line.

[0023] Furthermore, the step of generating a real-time avoidance solution based on the obstacle type and the real-time driving information includes:

[0024] Based on the DQN algorithm, the Q value of the driving path is estimated according to the obstacle type and the real-time driving information; wherein the DQN algorithm estimates the Q value of each possible action through a deep neural network, the deep neural network includes a main network and a target network, and the Q value includes a main Q value output by the main network and a target Q value of the path planned by the target network;

[0025] The real-time avoidance solution is generated according to the main Q value and the target Q value.

[0026] Furthermore, the step of estimating the Q value of the driving path according to the obstacle type and the real-time driving information includes:

[0027] generating classified safety data according to the obstacle type and the distance between the agent and the obstacle;

[0028] Input the classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane into the main network, and output the main Q value;

[0029] The classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane are input into the target network, and the target Q value is output.

[0030] Furthermore, the calculation formula of the main Q value is as follows:

[0031] Q(s,a) def =Q(s,a)+α*(R+γmax(Q(s′,a′))-Q(s,a))

[0032] Among them, s is the state, a is the action, s′ is the next state, a′ is the optimal action, α is the learning rate, r is the reward, and γ is the discount factor;

[0033] The calculation formula of the target Q value is as follows:

[0034] TargetQ(s,a) def =R+γ*max(Q_target(s′,a′))

[0035] Among them, Q_targrt is the Q value of the target network.

[0036] A multi-dimensional perception automatic driving avoidance device, comprising:

[0037] A data acquisition module is used to obtain real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road and three-dimensional point cloud data of the environment;

[0038] A feature extraction module, used to determine corresponding real-time obstacle features on the driving road according to the real-time perception data;

[0039] An obstacle classification module, used to determine the corresponding obstacle type according to the real-time obstacle characteristics;

[0040] A data processing module, configured to generate real-time driving information according to the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes a distance between the intelligent agent and the obstacle and a distance between the intelligent agent and the center line of the lane;

[0041] A path planning module is used to generate a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.

[0042] A device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the multi-dimensional perception autonomous driving avoidance method as described above are implemented.

[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-dimensional perception autonomous driving avoidance method as described above.

[0044] This application has the following advantages:

[0045] In the embodiments of the present application, relative to the problems in the prior art of "low detection accuracy, greatly affected by ambient light, and certain limitations in real-time, accuracy and robustness", the present application provides a solution to a multi-dimensional perception autonomous driving avoidance method, specifically: obtaining real-time perception data of an intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, thermal energy distribution data on the road, and three-dimensional point cloud data of the environment; determining corresponding real-time obstacle features on the driving road based on the real-time perception data; determining corresponding obstacle types based on the real-time obstacle features; generating real-time driving information based on the real-time obstacle features and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane; generating a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, deceleration driving, lane change driving, and parking. The present invention improves the accuracy and robustness of obstacle detection by comprehensively sensing obstacles on the road and comprehensively analyzing the sensing data from different sensors. It also analyzes the data in real time to achieve real-time monitoring and reporting of obstacles on the road, and can provide safe avoidance routes based on the detected obstacle information, thereby improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the description of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 This is a flowchart of the steps of a multi-dimensional perception autonomous driving avoidance method provided by an embodiment of the present application;

[0048] Figure 2 It is a flowchart of a multi-dimensional perception automatic driving avoidance method provided by an embodiment of the present application;

[0049] Figure 3 It is a structural block diagram of a multi-dimensional perception automatic driving avoidance device provided in one embodiment of the present application;

[0050] Figure 4 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the objects, features and advantages of the present application more obvious and understandable, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0052] Reference Figure 1 and Figure 2 , showing a multi-dimensional perception automatic driving avoidance method provided by an embodiment of the present application;

[0053] The method comprises:

[0054] S110, acquiring real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road, and three-dimensional point cloud data of the environment;

[0055] S120, determining corresponding real-time obstacle features on the driving road according to the real-time perception data;

[0056] S130, determining a corresponding obstacle type according to the real-time obstacle feature;

[0057] S140, generating real-time driving information according to the real-time obstacle feature and the real-time perception data; wherein the real-time driving information includes a distance between the intelligent agent and the obstacle and a distance between the intelligent agent and a center line of a lane;

[0058] S150. Generate a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.

[0059] In the embodiments of the present application, relative to the problems in the prior art of "low detection accuracy, greatly affected by ambient light, and certain limitations in real-time, accuracy and robustness", the present application provides a solution to a multi-dimensional perception autonomous driving avoidance method, specifically: obtaining real-time perception data of an intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, thermal energy distribution data on the road, and three-dimensional point cloud data of the environment; determining corresponding real-time obstacle features on the driving road based on the real-time perception data; determining corresponding obstacle types based on the real-time obstacle features; generating real-time driving information based on the real-time obstacle features and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane; generating a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, deceleration driving, lane change driving, and parking. The present invention improves the accuracy and robustness of obstacle detection by comprehensively sensing obstacles on the road and comprehensively analyzing the sensing data from different sensors. It also analyzes the data in real time to achieve real-time monitoring and reporting of obstacles on the road, and can provide safe avoidance routes based on the detected obstacle information, thereby improving driving safety.

[0060] Below, a multi-dimensional perception autonomous driving avoidance method in this exemplary embodiment will be further explained.

[0061] As described in step S110, real-time perception data of the intelligent body during driving is obtained; wherein the real-time perception data includes intelligent body driving data, road visual information, heat energy distribution data on the road and three-dimensional point cloud data of the environment.

[0062] It should be noted that this embodiment uses a variety of hardware to obtain multi-dimensional real-time perception data. These sensors are responsible for sensing different physical information, such as images, distances, speeds, etc. By comprehensively utilizing the data of these sensors, more comprehensive and accurate road obstacle information can be obtained.

[0063] The hardware used includes: infrared thermal imaging module, lidar module and various sensors.

[0064] Infrared thermal imaging module: It consists of an infrared thermal imager, an optical lens, an image sensor, a signal processor, and an interface circuit. The infrared thermal imaging module uses the infrared radiation emitted by an object to generate a thermal image. By measuring and recording the surface temperature of different areas of the object, it can display the thermal distribution of the object in real time. It can work in different environments and lighting conditions and is not affected by visible light. Using infrared thermal imaging technology, it can sense and capture the distribution of thermal energy on the road. The module uses an infrared camera or thermal imaging sensor to obtain thermal energy information on the road in real time and convert it into a digital signal for subsequent processing.

[0065] Lidar module: The Lidar module can provide high-precision three-dimensional perception data for obstacle detection and identification. It uses laser beams to scan the surrounding environment, measure and record information such as the distance, shape and position of obstacles, and is used for tasks such as obstacle detection, map construction and target tracking. The Lidar module obtains three-dimensional point cloud data of the road environment by emitting laser beams and receiving reflected signals. Then, by filtering, clustering and feature extraction of the point cloud data, obstacles on the road are identified, classified and tracked. Finally, corresponding alarm signals or control instructions are generated based on the detection results for applications such as driving assistance and traffic safety.

[0066] Sensors include cameras, radars, ultrasonic sensors, and inertial measurement units (IMUs).

[0067] Camera: used to obtain image information of the road and environment. Through image processing and computer vision algorithms, it can detect obstacles such as road signs, lane information, traffic lights, pedestrians, vehicles, etc., and provide visual perception capabilities.

[0068] Radar: Radar can transmit radio waves and receive their reflected signals to measure the distance, speed and direction of targets.

[0069] Ultrasonic sensor: sends ultrasonic pulses and measures their return time to determine the distance, usually used for close-range obstacle detection.

[0070] Inertial measurement unit: It consists of an accelerometer and a gyroscope, and is used to measure the acceleration and angular velocity of the device. Through integration processing, information such as the device's attitude, position, and motion state can be obtained.

[0071] The infrared thermal imaging module is used to obtain infrared images of road scenes, and the laser radar module is used to obtain three-dimensional point cloud data of road scenes. Sensors include but are not limited to high-definition cameras, millimeter-wave radars, laser radars and ultrasonic sensors. High-definition cameras can capture visual information of the road environment for subsequent image processing and obstacle identification. Millimeter-wave radars and laser radars can provide accurate data on obstacle distance, speed and direction. Ultrasonic sensors can achieve high-precision obstacle detection within a short distance. The present invention uses a variety of sensors and multi-dimensional perception data for comprehensive analysis to achieve accurate, real-time and comprehensive detection of obstacles on the road, providing drivers and traffic management departments with reliable road safety information.

[0072] As an example, multiple hardware devices are used to obtain multi-dimensional perception data, and navigation map data information can also be obtained from the navigation map database, including road network, traffic signs, traffic lights and other information. The user can also set the starting point and target location through the user interface or other means. After receiving the above information, the automatic driving avoidance device of the present invention uses it as input for avoiding scheme calculation.

[0073] In a specific implementation, the present application is also provided with a data storage and management module, which can persistently store the multi-dimensional perception data collected by the sensor to ensure the integrity and availability of the data. The module provides efficient data indexing and query functions, allowing researchers to easily retrieve and obtain data for a specific time period, a specific road section or a specific type of obstacle. The module supports the addition, deletion and modification of data to facilitate data cleaning and maintenance. At the same time, the module can also back up and restore data to ensure the security and reliability of the data. The module provides data analysis tools and algorithm interfaces for research work such as data mining, trend analysis and pattern recognition. Researchers can use this module to conduct in-depth analysis of long-term collected data to discover potential laws and trends. The module supports data visualization functions and can display data in the form of charts, curves, etc., so that researchers can understand and analyze data more intuitively.

[0074] As described in step S120, the corresponding real-time obstacle features on the driving road are determined based on the real-time perception data.

[0075] As an example, the data generated by the sensor is received, processed and analyzed by the real-time data processing module. The real-time data processing module can receive perception data from a variety of sensors, including cameras, radars, lidars and ultrasonic sensors, etc. These data include multi-dimensional information such as images, distances, and speeds. The real-time data processing module pre-processes the perception data from different sensors, including operations such as noise removal, filtering, and data smoothing, to achieve real-time monitoring and reporting of obstacles on the road to improve the quality and accuracy of the data. The collected raw data is pre-processed to remove noise and interference information to improve data quality. In this step, the road obstacle detection device pre-processes the collected raw data by filtering, Gaussian blurring, denoising, correcting, and converting the image to Lab color space to remove noise, filtering, and other interference information from the data collected by the sensor, thereby improving data quality and accuracy. For the real-time perception data of each sensor, the corresponding feature representation needs to be extracted. This can include image features, distance features, speed features, etc. The goal of feature extraction is to extract information that can describe the attributes of obstacles from the raw data. Deep learning can be used to extract the characteristic information of obstacles from the preprocessed data. The characteristic information includes the location, shape, size, texture, etc. of the object. By extracting the characteristic information, the obstacle recognition ability can be further enhanced.

[0076] As described in step S130, the corresponding obstacle type is determined according to the real-time obstacle feature.

[0077] As an example, after obtaining the feature representation of each sensor, in order to improve the accuracy of obstacle detection and classification, the perception data of different sensors needs to be fused. Since different sensors may have different sampling rates, coordinate systems and data formats, it is first necessary to align the perception data from different sensors. This embodiment is implemented through technologies such as timestamp matching and coordinate conversion, so that the data can be compared and fused at the same time and coordinate system. Before data fusion, the perception data from different sensors are preprocessed, and the corresponding feature representation is extracted for the perception data of each sensor. After obtaining the feature representation of each sensor, the data fusion algorithm will fuse these features. The fusion can be performed in a variety of ways, such as weighted averaging, decision cascade, probability fusion, etc.

[0078] The real-time data processing module uses pre-set algorithms and thresholds to detect obstacles on the road in real time. It can determine whether there are obstacles, locate and track them based on the features and patterns in the sensor data. When an obstacle is detected, the real-time data processing module can generate real-time reports and alarms. These reports and alarms can promptly notify drivers and traffic management departments so that they can take appropriate measures. When an obstacle is detected, the obstacle classification module classifies the detected obstacle to obtain the obstacle type corresponding to the obstacle. Specifically, using machine learning and computer vision algorithms, based on the extracted feature information, the classifier classifies and identifies the detected obstacles, distinguishes different types of obstacles, such as vehicles, pedestrians, bicycles, etc., and accurately classifies and identifies the detected obstacles based on the sensor data, thereby providing an important basis for subsequent response strategies. Among them, the classifier can be a model based on machine learning, such as support vector machine (SVM), random forest (Random Forest) or convolutional neural network (CNN). By accurately classifying obstacles, important information can be provided for subsequent decisions and behaviors.

[0079] As described in step S140, real-time driving information is generated based on the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent body and the obstacle and the distance between the intelligent body and the center line of the lane.

[0080] In one embodiment of the present invention, the specific process of "generating real-time driving information based on the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent body and the obstacle and the distance between the intelligent body and the center line of the lane" in step S140 can be further explained in combination with the following description.

[0081] As described in the following steps, generating the distance between the agent and the obstacle according to the real-time obstacle characteristics and the real-time perception data;

[0082] Determine the lane centerline based on the real-time perception data as described in the following steps;

[0083] As described in the following steps, the distance between the intelligent agent and the center line of the lane is generated based on the center line of the lane and the real-time perception data.

[0084] In one embodiment of the present invention, the specific process of "generating the distance between the agent and the obstacle based on the real-time obstacle characteristics and the real-time perception data" can be further explained in combination with the following description.

[0085] As an example, when an obstacle is detected, the real-time data processing module calculates the distance between the obstacle and the intelligent body through the perception data collected by the camera, radar or lidar, and generates real-time reports and alarms.

[0086] In one embodiment of the present invention, the specific process of "determining the lane center line based on the real-time perception data" can be further explained in combination with the following description.

[0087] Performing morphological processing on the visual information as described in the following steps;

[0088] As described in the following steps, the visual information is segmented and corresponding weights are determined to determine the minimum bounding rectangle of each portion of the visual information;

[0089] As described in the following steps, the center points of the lines of each part of the minimum circumscribed rectangle are weighted and summed to obtain the lane center line.

[0090] Specifically, remove all colors except white from the preprocessed image, perform morphological processing, change the shape and size of objects in the image, and remove noise from the image. The image is divided into three parts: upper, middle, and lower, and the weight of each part of the image is determined. Then, the target object is found from the image through contour detection, sorted according to the contour area, and the minimum circumscribed rectangle of the contour is obtained. The center point of the lane is obtained by summing the center points of the lines of the three parts according to the weights.

[0091] In one embodiment of the present invention, the specific process of "generating the distance between the intelligent agent and the lane boundary based on the lane boundary and the real-time perception data" can be further explained in combination with the following description.

[0092] As an example, when a lane boundary is detected, the real-time data processing module calculates the distance between the agent and the lane boundary through the perception data collected by the camera, radar or lidar.

[0093] In one embodiment of the present invention, the specific process of "generating the distance between the intelligent agent and the center line of the lane based on the center line of the lane and the real-time perception data" can be further explained in combination with the following description.

[0094] As an example, when the center line of a lane is detected, the real-time data processing module calculates the distance between the agent and the center line of the lane through the perception data collected by the camera, radar or lidar. Since there may be certain safety hazards at the lane boundary, the distance between the agent and the center line of the lane is calculated to keep a certain distance between the agent and the lane boundary so that the agent can drive along the middle of the lane.

[0095] In one embodiment of the present invention, the specific process of "generating a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking" described in step S150 can be further explained in combination with the following description.

[0096] It should be noted that the results of road condition detection are taken into consideration when planning avoidance plans. These results describe information such as the location, type and status of obstacles on the road. Avoidance plans and navigation are provided based on the detected obstacle information, and safe driving routes are provided. By comprehensively considering obstacles and their dynamic changes, potential collision risks can be avoided and driving safety and efficiency can be improved.

[0097] As described in the following steps, based on the DQN algorithm, the Q value of the driving path is estimated according to the obstacle type and the real-time driving information; wherein the DQN algorithm estimates the Q value of each possible action through a deep neural network, the deep neural network includes a main network and a target network, and the Q value includes a main Q value output by the main network and a target Q value of the path planned by the target network;

[0098] As described in the following steps, the real-time avoidance solution is generated according to the main Q value and the target Q value.

[0099] In one embodiment of the present invention, the specific process of “estimating the Q value of the driving path according to the obstacle type and the real-time driving information” can be further explained in combination with the following description.

[0100] Generating classified safety data according to the obstacle type and the distance between the agent and the obstacle as described in the following steps;

[0101] As described in the following steps, the classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane are input into the main network, and the main Q value is output;

[0102] As described in the following steps, the classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane are input into the target network, and the target Q value is output.

[0103] As an example, classified safety data is used to estimate specific safety values ​​under different types of obstacles. The farther the agent is from the obstacle, the higher the safety value, and the closer the agent is to the obstacle, the lower the safety value. The closer the agent is to the middle line of the lane, the higher the safety value, and the farther the agent is from the middle line of the lane, the lower the safety value.

[0104] The classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane are input into the target network, and the main Q value of the main network planning path is output. The calculation formula of the main Q value of the main network planning path is as follows:

[0105] Q(s,a) def =Q(s,a)+α*(R+γmax(Q(s′,a′))-Q(s,a))

[0106] Among them, s is the state, a is the action, s′ is the next state, a′ is the optimal action, α is the learning rate, r is the reward, and γ is the discount factor.

[0107] The classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane are input into the target network, and the target Q value of the target network planning path is output. The calculation formula of the target Q value of the target network planning path is as follows:

[0108] TargetQ(s,a) def =R+γ*max(Q_target(s′,a′))

[0109] Among them, Q_targrt is the target Q value of the target network.

[0110] Then, the loss function is generated by the calculation formula of the main Q value of the main network planning path and the calculation formula of the target Q value of the target network planning path. The calculation formula of the loss function is as follows:

[0111] Loss=MSE(Q(s,a),TargetQ(s,a))

[0112] The path planning module will eventually output the calculated real-time avoidance plan. The real-time avoidance plan includes normal driving, decelerated driving, lane change, parking, left / right deviation, etc. The real-time avoidance plan can be passed to the intelligent driving system to achieve automatic driving or provide navigation guidance to the driver. When the distance between the intelligent agent and the obstacle is less than the preset threshold, the obstacle avoidance logic is triggered, and the intelligent agent can slow down, change lanes or stop, and adjust the driving direction of the intelligent agent to avoid a collision between the intelligent agent and the obstacle. When the distance between the intelligent agent and the obstacle is greater than the preset threshold, it drives normally. When the distance between the intelligent agent and the obstacle is greater than the preset threshold, but deviates from the center line of the lane, the intelligent agent deviates to the left / right so that the intelligent agent can drive along the center line of the lane.

[0113] As another example, when, in addition to using a variety of hardware devices to obtain multi-dimensional perception data, navigation map data information including road networks, traffic signs, traffic lights, and information on the starting point and destination set by the user is also obtained, the real-time avoidance plan can be generated based on the main Q value, the target Q value, the navigation map data information, and the starting point and target position. At this time, the real-time avoidance plan includes normal driving, deceleration, lane change and parking, as well as the coordinate points of the path, turn instructions, driving distance, estimated time, etc. This information can be transmitted to the intelligent driving system to achieve automatic driving or provide navigation guidance to the driver. For example, when an obstacle is detected in front, the vehicle should originally slow down or stop. Since it is about to change lanes, it can change lanes in advance without slowing down or stopping. For example, when driving in the rightmost lane, when an obstacle is detected in front, the vehicle should originally change lanes to the left. Since it is about to turn right, it can slow down or stop instead of changing lanes.

[0114] In one embodiment of the present application, the real-time avoidance plan and obstacle identification results are displayed through an output unit, including but not limited to a display screen, voice broadcast, etc. The road obstacle detection device displays the results of identification and classification through an output unit. By displaying the identification results, the driver or system operator can obtain information about road obstacles in a timely manner to make corresponding decisions and actions. The present invention is also equipped with a user interface for displaying real-time road obstacle information to the driver, providing alarms and suggestions, and helping the driver make correct decisions. The user interface displays real-time road obstacle information to the driver, and the driver can view the detected obstacles through the interface and obtain relevant alarms and suggestions. The interface can display information such as the location, type and distance of the obstacle in a graphical manner, so that the driver can understand the road conditions more intuitively.

[0115] In one embodiment of the present application, the step also includes updating the perception data in real time and adaptively adjusting the perception parameters according to changes in the road environment and weather conditions. In this step, the road obstacle detection device updates the perception data in real time and adaptively adjusts the perception parameters according to changes in the road environment and weather conditions. For example, when the road conditions change or the weather is bad, the perception parameters can be adjusted by an adaptive algorithm to ensure the accuracy and stability of the detection. Such real-time updates and adaptive adjustments can improve the system's adaptability to different road environments.

[0116] The advantages of the present invention include but are not limited to the following aspects:

[0117] Multi-dimensional perception: The present invention uses a variety of sensors to obtain multi-dimensional perception data of the road environment, which can fully perceive obstacles on the road and improve the accuracy and reliability of detection.

[0118] Data fusion: The perception data from different sensors are comprehensively analyzed through the data fusion algorithm, making full use of the advantages of each sensor and improving the accuracy and robustness of obstacle detection.

[0119] Real-time: The present invention is equipped with a real-time data processing module, which can promptly receive, process and analyze the data generated by the sensors, realize real-time monitoring and reporting of obstacles on the road, and provide instant safety alerts and suggestions.

[0120] Efficiency: The efficient obstacle classification module can accurately classify and identify detected obstacles, provide detailed obstacle information to drivers and traffic management departments, and support targeted response strategies.

[0121] Path planning and navigation: The present invention is equipped with a path planning module, which can provide the driver with a safe avoidance plan based on the detected obstacle information, help the driver avoid obstacles and improve driving safety.

[0122] User-friendly interface: The present invention provides an intuitive and friendly user interface, which displays real-time road obstacle information to the driver, so that the driver can clearly understand the road conditions and make corresponding decisions.

[0123] Data storage and management: The present invention is equipped with a data storage and management module, which can effectively store and manage the collected road obstacle data, and provide support for long-term data analysis and trend prediction.

[0124] Wide application scenarios: The present invention can be applied to the fields of autonomous driving, intelligent transportation, or robot navigation, etc.

[0125] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0126] Reference Figure 3 , shows a multi-dimensional perception automatic driving avoidance device provided by an embodiment of the present application, including a central control module for controlling the operation of each module and outputting a planning result, and a data acquisition module, an image processing module, a data fusion module, a real-time data processing module, an obstacle classification module and a path planning module respectively connected to the central control module in communication; the image processing module is used to process and analyze image data, and the central control module is used to control the operation of each module and output an obstacle detection result;

[0127] The data acquisition module 310 is used to obtain the real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes the intelligent agent driving data, road visual information, heat energy distribution data on the road and three-dimensional point cloud data of the environment;

[0128] A feature extraction module 320, configured to determine corresponding real-time obstacle features on the driving road according to the real-time perception data;

[0129] The obstacle classification module 330 is used to determine the corresponding obstacle type according to the real-time obstacle characteristics;

[0130] A data processing module 340, configured to generate real-time driving information according to the real-time obstacle characteristics and the real-time perception data; wherein the real-time driving information includes a distance between the agent and the obstacle and a distance between the agent and the center line of the lane;

[0131] The path planning module 350 is used to generate a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.

[0132] In one embodiment of the present invention, the obstacle classification module 330 includes:

[0133] A fusion submodule, used for fusing the real-time obstacle features;

[0134] The classification submodule is used to determine the corresponding obstacle type according to the fused obstacle features.

[0135] In one embodiment of the present invention, the data processing module 340 includes:

[0136] The obstacle distance calculation submodule is used to generate the distance between the agent and the obstacle according to the real-time obstacle characteristics and the real-time perception data;

[0137] A lane centerline calculation submodule, used to determine the lane centerline based on the real-time perception data;

[0138] The lane centerline distance calculation submodule is used to generate the distance between the intelligent agent and the lane centerline based on the lane centerline and the real-time perception data.

[0139] In one embodiment of the present invention, the lane centerline calculation submodule includes:

[0140] A pre-processing unit, used for performing morphological processing on the visual information;

[0141] A weight unit, used to segment the visual information and determine corresponding weights, and determine a minimum bounding rectangle for each portion of the visual information;

[0142] The summing unit is used to perform weighted summing of the center points of the lines of each part of the minimum circumscribed rectangle to obtain the lane center line.

[0143] In one embodiment of the present invention, the path planning module 350 includes:

[0144] A Q-value calculation submodule is used to estimate the Q-value of the driving path based on the obstacle type and the real-time driving information based on a DQN algorithm; wherein the DQN algorithm estimates the Q-value of each possible action through a deep neural network, the deep neural network includes a main network and a target network, and the Q-value includes a main Q-value output by the main network and a target Q-value of the path planned by the target network;

[0145] The scheme generation submodule is used to generate the real-time avoidance scheme according to the main Q value and the target Q value.

[0146] In one embodiment of the present invention, the Q value calculation submodule includes:

[0147] A classification safety unit, used to generate classification safety data according to the obstacle type and the distance between the intelligent agent and the obstacle;

[0148] a main Q value calculation unit, configured to input the classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane into the main network, and output the main Q value;

[0149] The target Q value calculation unit is used to input the classified safety data, the distance between the intelligent agent and the obstacle, and the distance between the intelligent agent and the center line of the lane into the target network, and output the target Q value.

[0150] In one embodiment of the present invention,

[0151] The main Q value calculation unit comprises:

[0152] Q(s,a) def =Q(s,a)+α*(R+γmax(Q(s′,a′))-Q(s,a))

[0153] Among them, s is the state, a is the action, s′ is the next state, a′ is the optimal action, α is the learning rate, r is the reward, and γ is the discount factor;

[0154] The target Q value calculation unit comprises:

[0155] TargetQ(s,a) def =R+γ*max(Q_target(s′,a′))

[0156] Among them, Q_targrt is the Q value of the target network.

[0157] Reference Figure 4 , showing a computer device of a multi-dimensional perception automatic driving avoidance method of the present invention, which may specifically include the following:

[0158] The computer device 12 is in the form of a general-purpose computing device, and the components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0159] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor or a local bus 18 using any of a variety of bus 18 architectures. These architectures include, by way of example, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.

[0160] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0161] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42, which are configured to perform the functions of various embodiments of the present invention.

[0162] A program / utility 40 having a set (at least one) of program modules 42 may be stored in, for example, a memory, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0163] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, cameras, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs)), wide area networks (WANs), and / or public networks (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to: microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34, etc.

[0164] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a multi-dimensional perception automatic driving avoidance method provided in an embodiment of the present invention.

[0165] That is, when the processing unit 16 executes the program, it realizes: obtaining the real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes the intelligent agent driving data, road visual information, thermal energy distribution data on the road and three-dimensional point cloud data of the environment; determining the corresponding real-time obstacle features on the driving road according to the real-time perception data; determining the corresponding obstacle type according to the real-time obstacle features; generating real-time driving information according to the real-time obstacle features and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane; generating a real-time avoidance plan according to the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.

[0166] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multi-dimensional perception automatic driving avoidance method as provided in all embodiments of the present application:

[0167] That is, when the program is executed by the processor, it is implemented as follows: obtaining real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, thermal energy distribution data on the road and three-dimensional point cloud data of the environment; determining corresponding real-time obstacle features on the driving road based on the real-time perception data; determining corresponding obstacle types based on the real-time obstacle features; generating real-time driving information based on the real-time obstacle features and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane; generating a real-time avoidance plan based on the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.

[0168] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device.

[0169] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0170] The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0171] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0172] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0173] The above is a detailed introduction to the multi-dimensional perception automatic driving avoidance method and device provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multi-dimensional perception autonomous driving avoidance method, characterized in that: Includes steps: Acquire real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road and three-dimensional point cloud data of the environment; Determining corresponding real-time obstacle features on the driving road according to the real-time perception data; Determining a corresponding obstacle type according to the real-time obstacle feature; Generate real-time driving information based on the real-time obstacle features and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane; the center line of the lane is determined by the following steps: morphologically processing the road visual information; segmenting the road visual information and determining corresponding weights, determining the minimum bounding rectangle of each part of the road visual information; weighting the center points of the lines of each part of the minimum bounding rectangle to obtain the center line of the lane; Based on the DQN algorithm, a real-time avoidance plan is generated according to the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.

2. The method according to claim 1, characterized in that The step of determining the corresponding obstacle type according to the real-time obstacle feature comprises: fusing the real-time obstacle features; The corresponding obstacle type is determined according to the fused obstacle features.

3. The method according to claim 1, characterized in that The step of generating real-time driving information based on the real-time obstacle features and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane, includes: Generating a distance between the agent and the obstacle according to the real-time obstacle feature and the real-time perception data; Determining a lane centerline based on the real-time perception data; The distance between the agent and the center line of the lane is generated according to the center line of the lane and the real-time perception data.

4. The method according to claim 1, characterized in that: The step of generating a real-time avoidance solution based on the DQN algorithm and according to the obstacle type and the real-time driving information includes: Based on the DQN algorithm, the Q value of the driving path is estimated according to the obstacle type and the real-time driving information; wherein the DQN algorithm estimates the Q value of each possible action through a deep neural network, the deep neural network includes a main network and a target network, and the Q value includes a main Q value output by the main network and a target Q value of the path planned by the target network; The real-time avoidance solution is generated according to the main Q value and the target Q value.

5. The method according to claim 4, characterized in that The step of estimating the Q value of the driving path according to the obstacle type and the real-time driving information comprises: generating classified safety data according to the obstacle type and the distance between the agent and the obstacle; Input the classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane into the main network, and output the main Q value; The classified safety data, the distance between the agent and the obstacle, and the distance between the agent and the center line of the lane are input into the target network, and the target Q value is output.

6. The method according to claim 5, characterized in that The calculation formula of the main Q value is as follows: Q(s,a) def =Q(s,a)+α*(R+γmax(Q(s′,a′))-Q(s,a)) Among them, s is the state, a is the action, s′ is the next state, a′ is the optimal action, α is the learning rate, r is the reward, and γ is the discount factor; The calculation formula of the target Q value is as follows: TargetQ(s,a)def=R+γ*max(Q target(s′,a′)) Among them, Q_targrt is the target Q value.

7. A multi-dimensional perception automatic driving avoidance device, characterized in that: include: A data acquisition module is used to obtain real-time perception data of the intelligent agent during driving; wherein the real-time perception data includes intelligent agent driving data, road visual information, heat energy distribution data on the road and three-dimensional point cloud data of the environment; A feature extraction module, used to determine corresponding real-time obstacle features on the driving road according to the real-time perception data; An obstacle classification module, used to determine the corresponding obstacle type according to the real-time obstacle characteristics; A data processing module, for generating real-time driving information according to the real-time obstacle features and the real-time perception data; wherein the real-time driving information includes the distance between the intelligent agent and the obstacle and the distance between the intelligent agent and the center line of the lane; the center line of the lane is determined by the following steps: morphologically processing the road visual information; segmenting the road visual information and determining the corresponding weights, determining the minimum bounding rectangle of each part of the road visual information; weighted summing the center points of the lines of each part of the minimum bounding rectangle to obtain the center line of the lane; A path planning module is used to generate a real-time avoidance plan based on a DQN algorithm and in accordance with the obstacle type and the real-time driving information; wherein the real-time avoidance plan includes normal driving, decelerated driving, lane changing and parking.

8. A device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 6 when executed by the processor.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images