Unmanned driving control system and method based on multi-sensor coupling

Abstract

The invention relates to the technical field of unmanned driving, and discloses an unmanned driving control system and method based on multi-sensor coupling, and a sensing module of the system is integrated with a laser radar, a pulse vision camera and a millimeter wave radar, and is arranged at a plurality of preset positions of an unmanned vehicle. The environment sensing module is used for fusing information acquired by the laser radar, the pulse vision camera and the millimeter wave radar to obtain environment sensing data in the operation process of the unmanned vehicle; the positioning module obtains geographical location information of the unmanned vehicle; the decision planning module obtains information obtained by the sensing module and the positioning module through the communication module to generate decision information, generates corresponding control quantity based on the decision information and sends the control quantity to the operation control module for vehicle control. According to the invention, the environment perception precision of the unmanned vehicle is effectively improved, so that the decision planning module can make more accurate decision information to improve the stability and reliability of the unmanned system.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and in particular to an unmanned driving control system and method based on multi-sensor coupling. Background Art

[0002] The perception of the surrounding environment by unmanned vehicles depends on the sensors carried by the vehicle. Different types of sensors have different characteristics due to their different working principles and are suitable for different tasks. Since each sensor has its limitations. Using a single sensor for environmental perception can no longer meet the needs of unmanned vehicles for environmental perception accuracy. The use of multi-sensor fusion can make different types of sensors complement each other, and the use of different types of sensors can provide the unmanned driving system with an additional redundancy when a certain sensor fails for special reasons, ensuring that even if a certain sensor fails, some functions in emergency situations can still be retained, giving the unmanned driving system a certain warning. The data information generated by each sensor is inevitably mixed with noise, and the information of different sensors is difficult to simply mix together. When integrating the information list output by the sensor, there are often problems of mismatching due to information loss and information conflict.

[0003] The automatic obstacle avoidance technology of driverless cars still has the following problems: (1) Sensor data has interference factors such as noise, occlusion, and distortion, which leads to inaccurate detection of moving obstacles; (2) Data analysis and model prediction have uncertainties and errors, which leads to unreliable prediction of moving obstacle collision trajectories; (3) Intelligent decision-making and path planning have complexity and limitations, which leads to inefficient moving obstacle avoidance. In addition to the problem of low detection accuracy caused by its own transmission problems, rain, snow, dust, etc. in harsh environments will affect the accuracy of sensing elements and cause false detection problems. Summary of the invention

[0004] In view of this, the present invention provides an unmanned driving control system and method based on multi-sensor coupling to solve the problem of how to better obtain accurate environmental perception data for accurate control of unmanned vehicles.

[0005] In a first aspect, the present invention provides an unmanned driving control system based on multi-sensor coupling, comprising: a perception module, a positioning module, a communication module, a decision-making and planning module, and an operation control module;

[0006] The perception module integrates laser radar, pulse vision camera, and millimeter wave radar, and is set at multiple preset positions of the unmanned vehicle to integrate the information obtained by the laser radar, pulse vision camera, and millimeter wave radar to obtain environmental perception data during the operation of the unmanned vehicle;

[0007] A positioning module is used to obtain the geographic location information of the unmanned vehicle;

[0008] The decision planning module is used to obtain the information obtained by the perception module and the positioning module through the communication module to generate decision information, and generate corresponding control quantities based on the decision information and send them to the operation control module for vehicle control.

[0009] The unmanned driving control system based on multi-sensor coupling provided by the embodiment of the present invention has a perception module that integrates the respective advantages of three sensors: lidar, pulse vision camera and millimeter wave radar. The lidar can provide high-precision three-dimensional point cloud data and accurately depict the spatial position and outline of objects around the vehicle. The pulse vision camera can capture rich visual information. The millimeter wave radar performs well in detecting the distance, speed and angle of objects. Setting these sensors at multiple preset positions of the unmanned vehicle can further expand the perception range and effectively improve the environmental perception accuracy of the unmanned vehicle, which is conducive to the decision-making planning module to make more accurate decision information to improve the stability and reliability of the unmanned driving system.

[0010] In an optional implementation, the perception module is disposed at least at four vehicle corners on the outside of the unmanned vehicle.

[0011] The embodiment of the present invention sets perception modules at the four corners of the outside of the unmanned vehicle, which can minimize the perception blind spots around the vehicle. Sensors at different corners have different scanning angles for objects. The scanning data at different angles are integrated to provide more accurate environmental perception data for vehicle decision-making planning.

[0012] In a second aspect, the present invention provides an unmanned driving control method based on multi-sensor coupling, comprising:

[0013] The data obtained by the laser radar, pulse vision camera, and millimeter wave radar of the same scene are unified and then fused to obtain environmental perception data;

[0014] Decision information is generated based on the environmental perception data and the vehicle's geographic location information, and the corresponding control amount is generated based on the decision information to control the vehicle.

[0015] The embodiment of the present invention is based on the unmanned driving control method of multi-sensor coupling. The laser radar can provide high-precision three-dimensional point cloud data to accurately depict the spatial position and outline of objects around the vehicle. The pulse vision camera can capture rich visual information. The millimeter wave radar performs well in detecting the distance, speed and angle of objects. By fusing the data of these three sensors, the environmental perception accuracy of the unmanned vehicle is effectively improved, which is conducive to the decision-making planning module to make more accurate decision information to improve the stability and reliability of the unmanned driving system.

[0016] In an optional implementation, the environmental data perception unit unifies the coordinates of the data acquired by the laser radar, pulse vision camera, and millimeter wave radar for acquiring the same scene and then performs data fusion, including:

[0017] Based on the vehicle coordinate system, the data of LiDAR, pulse vision camera and millimeter wave radar are unified into a common reference coordinate system;

[0018] Initial weights are set for the lidar, pulse vision camera and millimeter-wave radar, and the data of these three sensors are processed for variance respectively. The weight allocation algorithm based on the variance mean deviation and the weight allocation algorithm based on the variance mean change rate together form an automatic weight allocation algorithm for data fusion.

[0019] The automatic weight allocation algorithm of the embodiment of the present invention, which is composed of the weight allocation algorithm of the variance mean deviation and the weight allocation algorithm of the variance mean change rate, comprehensively considers the two important aspects of data stability (reflected by the variance mean deviation) and adaptability to environmental changes (reflected by the variance mean change rate). This combination method can dynamically and flexibly adjust the weight of each sensor in data fusion according to actual environmental conditions, sensor working status and other factors, so that the environmental perception data finally obtained by fusion is more accurate and reliable, and can better adapt to complex and changeable traffic scenes and various working conditions during vehicle driving, thereby laying a solid foundation for unmanned vehicles to make reasonable decisions and safety control based on accurate environmental perception.

[0020] In an optional implementation, the process of fusing data includes:

[0021] Processing the point cloud data within a preset range around the vehicle acquired by the laser radar to obtain the spatial position, object contour information and direction information of objects in the vehicle's surrounding environment as the first perception information;

[0022] Process and recognize the images acquired by the pulse vision camera to obtain traffic scene layout information, lighting and weather information, traffic signs and signal information, and dynamic behavior information around the vehicle as the second perception information;

[0023] The data acquired by the millimeter-wave radar is processed to obtain the distance, speed and angle of objects in the vehicle's surrounding environment as the third perception information;

[0024] The first perception information, the second perception information, and the third perception information are fused to obtain environmental perception data.

[0025] The embodiments of the present invention integrate space and visual details and coordinate dynamic and static information to make the vehicle's perception of environmental objects more complete and accurate, which can better control the safety and smoothness of the operation of the unmanned vehicle and reduce the probability of traffic accidents.

[0026] In an optional embodiment, the decision information is generated based on the environmental perception data and the geographic location information of the vehicle, including: global planning and local planning of the vehicle, the global planning generates a global planning path based on the vehicle positioning information and the departure location and the target location; the local planning performs local path planning on the vehicle based on the global planning, using the road boundary condition constraints in the environmental perception data, the obstacle boundary constraints around the vehicle, and the vehicle dynamics constraints and kinematic constraints.

[0027] The global planning of the embodiment of the present invention generates a path based on the vehicle positioning information as well as the departure point and destination, providing a macro driving strategy for the vehicle. On the basis of the global planning, the local planning considers various boundary conditions in the environmental perception data and the vehicle's own constraints. This is a detailed planning at the micro level. The planning method that combines the macro and micro levels makes the vehicle's driving path meet the overall goal and can flexibly respond to various specific road conditions.

[0028] In an optional embodiment, the method of generating a corresponding control amount based on the decision information to control the vehicle includes: analyzing the global plan and the local plan, preliminarily determining the range and value of the control amount, and while the vehicle is driving according to the preliminarily determined control amount, continuously collecting environmental perception data and the vehicle's own real-time status information through various sensors to compare and analyze the decision to generate feedback information, and optimizing the control amount based on the feedback information.

[0029] During vehicle driving, the embodiments of the present invention rely on various sensors to continuously collect environmental perception data and the vehicle's own real-time status information, and compare and analyze the data with the decision-making information to generate feedback information, so as to grasp the difference between the actual driving conditions of the vehicle and the expected conditions in real time. When a sensor in the system has a temporary abnormality or data error, reliable vehicle control can still be maintained through comprehensive comparison of multi-source information and feedback adjustment, thereby reducing the risk of failure of the entire system due to local failures and improving the overall reliability of the unmanned driving system.

[0030] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the unmanned driving control method based on multi-sensor coupling of the above-mentioned second aspect or any corresponding embodiment thereof.

[0031] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the unmanned driving control method based on multi-sensor coupling of the above-mentioned second aspect or any corresponding embodiment thereof.

[0032] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the unmanned driving control method based on multi-sensor coupling of the above-mentioned second aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 is a structural block diagram of an unmanned driving control system based on multi-sensor coupling according to an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of the deployment position of the perception module on a vehicle according to an embodiment of the present invention;

[0036] Figure 3 is a flow chart of an unmanned driving control method based on multi-sensor coupling according to an embodiment of the present invention;

[0037] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0039] At present, the more mature driverless cars use multiple sets of perception systems, including 64-line laser radar, blind spot radar, high-definition camera and millimeter wave radar. High-definition camera can be used for obstacle detection, lane line detection, road sign recognition, etc.; laser radar can obtain the 3D model of the car's surrounding environment, and can compare the global map obtained in real time with the features in the high-precision map, so as to achieve navigation and enhance the positioning accuracy of the vehicle. However, millimeter wave radar may be interfered by other electronic devices or signals in the same frequency band when working, affecting its normal detection and data transmission, and reducing the reliability and stability of the perception system. The image quality and recognition effect of high-definition camera depend to a large extent on lighting conditions, and the recognition and detection accuracy of objects will be affected in the case of strong direct light, backlight, weak light or uneven light.

[0040] In order to improve the accuracy and anti-interference ability of the environment perception of the driverless car. This embodiment provides an unmanned driving control system based on multi-sensor coupling, which is applied to the driverless car as a L4 level autonomous driving vehicle, which can achieve fully reliable autonomous driving on open roads and can cope with typical road elements such as dynamic obstacles, including autonomous decision-making, dynamic path planning, lane keeping, adaptive cruise, autonomous lane changing, autonomous overtaking and other functions.

[0041] like Figure 1 As shown, the system includes: a perception module, a positioning module, a communication module, a decision-making and planning module and an operation control module; wherein the perception module integrates a laser radar, a pulse vision camera, and a millimeter-wave radar, and is set at multiple preset positions of the unmanned vehicle, and is used to fuse the information obtained by the laser radar, the pulse vision camera, and the millimeter-wave radar to obtain the environmental perception data of the operation process of the unmanned vehicle; the positioning module is used to obtain the geographic location information of the unmanned vehicle; the decision-making and planning module is used to obtain the information obtained by the perception module and the positioning module through the communication module to generate decision information, and generate corresponding control quantities based on the decision information and send them to the operation control module for vehicle control.

[0042] Specifically, the laser radar of the perception module uses visible light and near-infrared light waves to emit, transmit and receive to detect objects. The laser radar has two core functions in unmanned driving: 1. Environmental perception through three-dimensional modeling, through laser scanning to obtain a 3D model of the car's surroundings, and use relevant algorithms to compare the changes in the environment between the previous frame and the next frame to detect the surrounding environment; 2. Enhance positioning through synchronous mapping, 3D laser radar compares the global map obtained in real time with the high-precision map features, thereby achieving navigation and enhancing the positioning accuracy of the vehicle. Millimeter-wave radar determines the physical environment information around the car body (such as the relative distance, relative speed, angle, direction of movement, etc. between the vehicle and other objects) by emitting radio signals (electromagnetic waves in the millimeter wave band) and receiving reflected signals. Millimeter-wave radar tracks and identifies and classifies targets based on the detected object information, and then combines the dynamic information of the vehicle body for data fusion to make reasonable decisions and reduce the probability of accidents.

[0043] The embodiment of the present invention uses a pulse vision camera as a visual sensor. Each pixel on the pulse vision camera sensor independently and continuously captures incoming photons, and triggers a pulse only when the accumulated photons reach a certain threshold. Therefore, a continuous pulse stream can be generated with a very high time resolution, and instantaneous scene changes can be captured. For fast-moving objects or fast-changing scenes, such as obstacles that suddenly appear around the vehicle when driving at high speed, it can be more timely and accurately perceived and recorded, providing more timely and effective information for the unmanned driving system to make fast and accurate decisions. It has the characteristics of free dynamic range. Unlike traditional cameras that are limited to the exposure time window, it can better capture the details of bright and dark parts at the same time in high-light-ratio scenes, such as environments where strong light and shadows coexist, without overexposure or underexposure, so as to perceive the surrounding environment more comprehensively and accurately. Due to its unique sensing method and high time resolution, the pulse vision camera sensor can quickly convert light signals into electrical pulse signals and transmit and process them. Compared with traditional cameras, the data acquisition and transmission delay is lower, so that the unmanned driving system can receive the latest environmental information faster, further improve the timeliness and accuracy of decision-making, and respond to emergencies more quickly.

[0044] When the pulse vision camera sensor is working, it will trigger a pulse and transmit data only when the accumulated photons of the pixel reach the threshold. Compared with the continuous exposure and data transmission mode of traditional cameras, its power consumption is greatly reduced. For unmanned vehicles, this helps to reduce overall energy consumption, extend the vehicle's cruising range, and improve energy efficiency. It is especially important for unmanned driving platforms such as electric vehicles that have high requirements for cruising range. The use of pulse vision cameras can provide unmanned driving systems with richer environmental perception data, help to more accurately identify and locate obstacles, judge the distance relationship between vehicles and surrounding objects, etc., and also provide the possibility of reducing system costs and complexity; traditional cameras are prone to motion blur when shooting high-speed moving objects, affecting image quality and subsequent recognition processing. The working principle of the pulse vision camera sensor makes it capture moving objects more clearly and accurately, is not easily affected by motion blur, and can better identify and track moving vehicles, pedestrians and other objects, providing more reliable protection for the safe driving of unmanned driving.

[0045] In addition to the traditional installation of blind spot radar on unmanned vehicles, in the embodiment of the present invention, laser radar, millimeter wave radar, and pulse vision camera are integrated into a perception module, such as Figure 2 As shown, the sensors are set at the four corners of the unmanned vehicle to sense the environmental changes during the operation of the vehicle in real time. The sensors at different corners have different scanning angles for objects. The fusion of the scanning data at different angles can provide more accurate environmental perception data for the vehicle's decision-making planning. Through four sets of perception modules, 360-degree blind spot-free perception can be achieved, and cubes with a side length of not less than 15 cm, as well as various dynamic objects and obstacles within 250 meters, can be stably detected, effectively improving the environmental perception accuracy of unmanned vehicles, thereby facilitating the decision-making planning module to make more accurate decision information to improve the stability and reliability of the unmanned driving system. In actual applications, the number of settings can also be increased at other locations based on the four corners of the vehicle according to needs, and there is no limit here.

[0046] The positioning module in the embodiment of the present invention can be a relatively mature positioning system such as Beidou, GPS, etc., and the communication module can be an in-vehicle Ethernet, which is an in-vehicle communication network based on Ethernet technology, using physical media such as twisted pair cables, and can provide a transmission rate of up to 10Gbps or even higher. This can meet the high-speed transmission requirements of large amounts of data in the unmanned driving perception module. This is only an example and is not limited to this.

[0047] Based on the above-mentioned control system, an embodiment of the present invention also provides an embodiment of an unmanned driving control method based on multi-sensor coupling. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here. Figure 3 is a flow chart of an unmanned driving control method based on multi-sensor coupling according to an embodiment of the present invention, such as Figure 3 As shown, the process includes the following steps:

[0048] S101, unify the coordinates of the data obtained by the laser radar, pulse vision camera, and millimeter wave radar of the same scene, and then fuse the data to obtain environmental perception data. Specifically, the following steps are included:

[0049] a1, based on the vehicle coordinate system, the data of LiDAR, pulse vision camera and millimeter wave radar are unified into a common reference coordinate system;

[0050] Specifically, the laser radar usually establishes a coordinate system with its own installation position as the origin. Generally, its Z axis is vertically upward, the X axis is forward (along the main scanning direction of the laser beam emitted by the laser radar), and the Y axis is determined according to the right-hand rule. It can obtain three-dimensional point cloud data of the surrounding environment by scanning, reflecting the spatial position information of the object relative to the laser radar itself; the pulse vision camera coordinate system also takes its own installation position as the origin, and the definition of its coordinate axis direction is related to the camera imaging principle. For example, the X-axis and Y-axis determine the two-dimensional coordinate direction of the imaging plane, and mainly capture visual information in the environment by taking images, but the image information is initially based on The coordinate system of the camera itself only reflects the relative position and appearance characteristics of the object in the camera's view. The coordinate system of the millimeter-wave radar is also constructed based on its own installation position. Generally, the X-axis points to the detection direction of the radar (usually the set direction such as the front of the vehicle), the Z-axis is vertically upward, and the Y-axis is determined by the right-hand rule. It mainly relies on transmitting and receiving millimeter-wave signals to detect the distance, angle, speed and other information of the target object. This information is expressed based on the coordinate system of the millimeter-wave radar itself; the vehicle coordinate system usually takes the center of mass of the vehicle as the origin, the X-axis points to the positive direction of the vehicle's forward movement, the Y-axis points to the left side of the vehicle, and the Z-axis is vertically upward. This coordinate system provides a common reference framework for unifying the data of various sensors. All subsequent sensor data must be converted to this coordinate system so that the vehicle control system can comprehensively use these data from different sensors for accurate environmental perception, decision-making and control.

[0051] When performing coordinate conversion, it is necessary to accurately measure the installation position and posture of each sensor on the vehicle (including pitch angle, yaw angle, roll angle and other angular information), and determine its relative relationship with the vehicle coordinate system by measuring its translation vector relative to the center of mass of the vehicle (that is, the three-dimensional coordinate difference of the installation position) and the installation posture angle. Such precise measurement and positioning work is also required for pulse vision cameras and millimeter-wave radars. The mathematical principle of rigid body transformation can be used for coordinate conversion. Through coordinate conversion calculations like this, the coordinates of the data points obtained by the lidar, pulse vision camera, and millimeter-wave radar are converted from their original sensor coordinate systems to the vehicle coordinate system, so that all data can be represented in the same reference coordinate system, which is convenient for subsequent data fusion, target recognition and other operations.

[0052] a2, set initial weights for lidar, pulse vision camera and millimeter wave radar, and perform variance processing on the data of these three sensors respectively, and then use the weight allocation algorithm based on the mean deviation of variance and the weight allocation algorithm based on the mean rate of change of variance to form an automatic weight allocation algorithm for data fusion.

[0053] Specifically, the initial weight values ​​of the three are set to 0.3333, 0.3333, and 0.3334, respectively, which means that the data accuracy perceived by the three sensors is the same; by processing the variance of each sensor, the weight allocation algorithm considering the average deviation of the variance and the weight allocation algorithm considering the average change rate of the variance are used to form an automatic weight allocation algorithm (that is, the three sensor allocation conditions are divided into two parts: one is the weight allocation based on the average deviation of the variance; the other is the weight allocation based on the average change rate of the variance, and the sum of the weights of the two parts is used as the final weight allocation). The specific process is:

[0054] a21, calculate the average of the mean deviations of the three sensor variances:

[0055]

[0056] Where d is the average of the average deviations of the three sensors; d L is the average deviation of the lidar variance; d R Millimeter wave radar variance mean deviation; d C Pulse Vision Camera Variance Mean Deviation. Calculate the difference between the mean deviation of each sensor and the mean of the three sensor variance mean deviations:

[0057] D L =dd L

[0058] D R =dd R

[0059] DC =dd C

[0060] The above formulas are the differences between the average deviations of the laser radar, millimeter wave radar, and pulse vision camera and the mean of the average deviations of the three sensors, and then the weights are normalized, that is, the weight distribution:

[0061]

[0062] Among them, W L1 , W R1 , W C1 They are the lidar weight, millimeter wave radar weight and pulse vision camera weight assigned according to the variance mean deviation.

[0063] a22, weight distribution according to the average rate of change of variance:

[0064]

[0065] Among them, r L is the average deviation of the lidar variance; r R Millimeter wave radar variance mean deviation; r C Pulse Vision Camera Variance Mean Deviation.

[0066] a23, calculate the difference between the average deviation of each sensor and the mean deviation of the three sensor variances:

[0067] R L =rr L

[0068] R R =rr R

[0069] R C =rr C

[0070] The above formulas are the differences between the average deviations of the laser radar, millimeter wave radar, and pulse vision camera and the mean of the average deviations of the three sensors, and then the weights are normalized, that is, the weight distribution:

[0071]

[0072] Among them, W L2 , W R2 , W C2 They are the lidar weight, millimeter wave radar weight and pulse vision camera weight allocated according to the average rate of change of variance.

[0073] a24, add the two weights to get the final weight of each sensor:

[0074] W L=W L1 +W L2

[0075] W R =W R1 +W R2

[0076] W C =W C1 +W C2

[0077] Among them, W L , W R , W C The final weights assigned to the lidar weight, mmWave radar weight, and pulse vision camera, respectively.

[0078] It should be noted that the larger the variance mean deviation is, the greater the problem may be with the sensor, and the weight of the sensor will be reduced accordingly, or even zero. This effectively eliminates bad information when the same scene is perceived by three sensors. The automatic weight allocation algorithm for the variance change rate compares the variance of each frame object of the sensor with the variance of the previous frame object to obtain the variance change rate. Usually, when detecting the movement of an object, the variance change rate changes steadily. When the variance change rate is too large, it means that the change of the outlier variance point is too obvious, and the detection data of the point has problems, so the data of this point needs to be eliminated.

[0079] Furthermore, the process of fusing data in the embodiment of the present invention includes:

[0080] b1, processing the point cloud data within a preset range around the vehicle acquired by the laser radar to obtain the spatial position, object contour information and direction information of objects in the vehicle's surrounding environment as the first perception information;

[0081] b2, processing and identifying the images acquired by the pulse vision camera to obtain traffic scene layout information, lighting and weather information, traffic signs and signal information, and dynamic behavior information around the vehicle as the second perception information;

[0082] b3, processing the data acquired by the millimeter-wave radar to obtain the distance, speed and angle of objects in the vehicle's surrounding environment as the third perception information;

[0083] b4, perform data fusion on the first perception information, the second perception information, and the third perception information to obtain environmental perception data.

[0084] In the embodiment of the present invention, the pulse vision camera can directly obtain illumination and weather information, and it can provide rich visual information when the weather is good and the illumination is sufficient. However, in bad weather (such as heavy rain, dense fog) or poor illumination conditions (such as at night, strong light), the performance of the laser radar and millimeter wave radar is less affected by weather and illumination. They can continue to provide key environmental perception data, such as the position and distance of objects, when the quality of camera data decreases, so as to ensure that the entire system can operate stably in various environments. Due to the fusion of multiple sensor information, when a sensor fails or the data has errors, the data of other sensors can play a supplementary and corrective role. For example, if the image data of the pulse vision camera is temporarily interfered by noise, the data of the laser radar and millimeter wave radar can still provide the basic position and motion state information of the object, reducing the risk of system failure due to a single sensor problem. By combining the traffic scene layout information (second perception information) with the spatial position and contour information of the object (first perception information) and the distance, speed and angle of the object (third perception information), the system can build a complete traffic scene model. This includes a lot of information such as the geometry of the road, the distribution of lanes, the position and movement status of vehicles and pedestrians, and the location of traffic signs and signals, which provides a comprehensive scenario basis for the vehicle's autonomous driving decisions, can better control the safety and smoothness of unmanned vehicle operations, and reduce the probability of traffic accidents.

[0085] S102, generating decision information based on the environmental perception data and the geographic location information of the vehicle, and generating a corresponding control amount based on the decision information to control the vehicle.

[0086] The decision-making planning in the embodiment of the present invention can be divided into two levels: global planning and local planning. Among them, the task of global planning is to plan a path from the starting point to the target point based on the information of the global map database, but does not consider the detailed information of the path such as direction, width, curvature, roadblocks, etc. Therefore, during driving, it is more dependent on local planning of an ideal local path without collision. On the basis of global planning, local planning plans the local path of the vehicle based on the road boundary condition constraints in the environmental perception data, the boundary constraints of obstacles around the vehicle, and the vehicle dynamics constraints and kinematic constraints to ensure the rationality and safety of the vehicle in tracking control.

[0087] The local optimal path under global path planning should minimize the deviation from the global path as much as possible to ensure the rationality of the global planning path. For example, the artificial intelligence algorithm of deep learning is used to plan the path trajectory with the above constraints, and the target is recognized and tracked efficiently and accurately, so that the car can run smoothly during driving and avoid obstacles. The spline curve is constrained based on cost functions such as control amount smoothness or minimum deviation to obtain the best running trajectory.

[0088] The global planning of the embodiment of the present invention generates a path based on the vehicle positioning information as well as the departure point and destination, providing a macro driving strategy for the vehicle. On the basis of the global planning, the local planning considers various boundary conditions in the environmental perception data and the vehicle's own constraints. This is a detailed planning at the micro level. The planning method that combines the macro and micro levels makes the vehicle's driving path meet the overall goal and can flexibly respond to various specific road conditions.

[0089] In the embodiment of the present invention, the corresponding control amount is generated based on the decision information to control the vehicle, including: parsing the global plan and the local plan, preliminarily determining the range and value of the control amount, and when the vehicle is driving according to the preliminarily determined control amount, continuously collecting environmental perception data and the real-time status information of the vehicle through various sensors, comparing and analyzing the decision and generating feedback information, and optimizing the control amount based on the feedback information. The range and value of the control amount are preliminarily determined, for example:

[0090] 1. Steering wheel angle control: If the target path includes a curve, the appropriate steering wheel angle is calculated based on the curvature radius of the curve and the steering characteristics of the vehicle to ensure that the vehicle can pass the curve smoothly and stay in the lane. For example, a curve with a smaller curvature radius requires a larger steering wheel angle to achieve steering.

[0091] 2. Speed ​​control: Set the appropriate speed by considering factors such as the speed limit, the complexity of the road conditions (such as whether there are frequent pedestrians or other vehicles), and the requirements of different sections on the route. For example, in sections with lower speed limits such as school areas, the speed control will be set at a lower value; in smooth straight sections of highways, the speed control can be appropriately increased.

[0092] 3. Throttle and brake control: After the speed control is determined, the speed can be further increased by the throttle opening, or reduced by the brake force. When acceleration is required, the throttle opening is gradually increased to allow the vehicle to accelerate steadily; when approaching an obstacle, encountering a red light, or needing to slow down and turn, the brake force is reasonably applied to achieve deceleration and stop.

[0093] The above is only an example and is not limited to this. In the process of vehicle driving, the embodiment of the present invention relies on various sensors to continuously collect environmental perception data and the real-time status information of the vehicle itself, and compares and analyzes with the decision information to generate feedback information, which can grasp the difference between the actual driving situation of the vehicle and the expected situation in real time. When a sensor in the system has a short-term abnormality or data error, through the comprehensive comparison of multi-source information and feedback adjustment, reliable vehicle control can still be maintained, reducing the risk of failure of the entire system due to local failures, and improving the overall reliability of the unmanned driving system.

[0094] An embodiment of the present invention further provides a computer device for executing the unmanned driving control method based on multi-sensor coupling in the above embodiment.

[0095] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0096] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0097] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0098] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0099] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0100] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0101] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor central control system or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0102] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0103] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An unmanned driving control system based on multi-sensor coupling, characterized in that: include: Perception module, positioning module, communication module, decision-making and planning module, and operation control module; The perception module integrates laser radar, pulse vision camera, and millimeter wave radar, and is set at multiple preset positions of the unmanned vehicle to integrate the information obtained by the laser radar, pulse vision camera, and millimeter wave radar to obtain environmental perception data during the operation of the unmanned vehicle; A positioning module is used to obtain the geographic location information of the unmanned vehicle; The decision planning module is used to obtain the information obtained by the perception module and the positioning module through the communication module to generate decision information, and generate corresponding control quantities based on the decision information and send them to the operation control module for vehicle control.

2. The system according to claim 1, characterized in that The sensing module is at least arranged at four vehicle corners on the outside of the unmanned vehicle.

3. An unmanned driving control method based on multi-sensor coupling, based on the system according to claim 1 or 2, characterized in that: include: The data obtained by the laser radar, pulse vision camera, and millimeter wave radar of the same scene are unified and then fused to obtain environmental perception data; Decision information is generated based on the environmental perception data and the vehicle's geographic location information, and the corresponding control amount is generated based on the decision information to control the vehicle.

4. The method according to claim 3, characterized in that The data acquired by the laser radar, pulse vision camera, and millimeter wave radar for acquiring the same scene are unified in coordinates and then fused to obtain environmental perception data, including: Based on the vehicle coordinate system, the data of LiDAR, pulse vision camera and millimeter wave radar are unified into a common reference coordinate system; Initial weights are set for the lidar, pulse vision camera and millimeter-wave radar, and the data of these three sensors are processed for variance respectively. The weight allocation algorithm based on the variance mean deviation and the weight allocation algorithm based on the variance mean change rate together form an automatic weight allocation algorithm for data fusion.

5. The method according to claim 3, characterized in that: The process of data fusion includes: Processing the point cloud data within a preset range around the vehicle acquired by the laser radar to obtain the spatial position, object contour information and direction information of objects in the vehicle's surrounding environment as the first perception information; Process and recognize the images acquired by the pulse vision camera to obtain traffic scene layout information, lighting and weather information, traffic signs and signal information, and dynamic behavior information around the vehicle as the second perception information; The data acquired by the millimeter-wave radar is processed to obtain the distance, speed and angle of objects in the vehicle's surrounding environment as the third perception information; The first perception information, the second perception information, and the third perception information are fused to obtain environmental perception data.

6. The method according to claim 5, characterized in that The decision information is generated based on the environmental perception data and the geographic location information of the vehicle, including: global planning and local planning of the vehicle, wherein the global planning generates a global planning path based on the vehicle positioning information and the departure location and the target location; the local planning performs local path planning on the vehicle based on the global planning, using the road boundary condition constraints in the environmental perception data, the obstacle boundary constraints around the vehicle, and the vehicle dynamics constraints and kinematic constraints.

7. The method according to claim 6, characterized in that The method of generating a corresponding control amount based on the decision information to control the vehicle includes: analyzing the global plan and the local plan, preliminarily determining the range and value of the control amount, and while the vehicle is driving according to the preliminarily determined control amount, continuously collecting environmental perception data and the vehicle's own real-time status information through various sensors to compare and analyze the decision to generate feedback information, and optimizing the control amount based on the feedback information.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the unmanned driving control method based on multi-sensor coupling as described in any one of claims 3 to 7 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the unmanned driving control method based on multi-sensor coupling as described in any one of claims 3-7.

10. A computer program product, characterized in that It includes computer instructions, which are used to enable a computer to execute the unmanned driving control method based on multi-sensor coupling as described in any one of claims 3-7.

Images