Safety protection method for outdoor AGV in extreme weather based on multi-sensor fusion

Abstract

The invention belongs to the field of outdoor AGV safety protection, and particularly relates to a multi-sensor fusion outdoor AGV extreme weather safety protection method comprising the following steps: carrying out time synchronization and space synchronization on a laser radar, an infrared thermal imaging camera and a 4d millimeter wave lightning sensor arranged on an AGV; after original data acquisition is carried out on data of each sensor, sensor state abnormity judgment and data abnormity judgment are carried out, and abnormal data are processed; when the AGVs execute tasks and travel routes, the running states of the surrounding AGVs are obtained in real time through a scheduling system; performing feature-level and decision-level data fusion on the data of each sensor in combination with the running state of the AGV to obtain a safety protection area or obstacle existence information; performing rationality judgment on the range of the protection area, further calculating an AGV driving area, planning an AGV driving route and speed planning in the safety protection area, and controlling the AGV to operate; otherwise, AGV parking protection is controlled.

Description

Technical Field

[0001] The present invention belongs to the field of safety protection of outdoor AGV, and specifically is a multi-sensor fusion safety protection method for outdoor AGV under extreme weather conditions. Background Art

[0002] AGV safety protection is a prerequisite for the safe operation of AGV. Safety protection requires that during the operation of AGV, various sensors can be used to collect information about the surrounding environment and obstacles, and that the AGV can autonomously avoid obstacles and bypass obstacles according to certain strategies based on this information to ensure the safe, reliable and efficient operation of AGV. Safety protection requires obtaining information about the surrounding environment through sensors, including information such as the size, shape and position of obstacles. Generally speaking, the sensors used mainly include cameras, laser radars and contact safety anti-collision devices.

[0003] When faced with severe weather, such as rain, snow, fog, dust storms, etc., the sensors will be affected to varying degrees, reducing the perception accuracy and range. Rain, snowflakes or fog will form water droplets or frost on the camera lens, causing blurred images and affecting the recognition of the environment and obstacles. Raindrops or snowflakes can scatter the laser beam of the lidar, making it difficult for the lidar to penetrate the rain curtain or snow fog at a longer distance, causing the echo signal to weaken, reducing the effective measurement working range, and affecting the recognition and tracking of distant objects. In a dense fog environment, the lidar may only be able to detect objects at a close distance, and its long-distance perception capability is greatly reduced. In summary, extreme weather poses a great challenge to AGV safety protection and affects the accuracy and reliability of AGV decision-making. These are the problems faced by outdoor AGV safety protection. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-sensor fusion outdoor AGV extreme weather safety protection method, which greatly improves the robustness and accuracy of the safety protection of the outdoor AGV in abnormal weather (rain, snow, fog, etc.), ensuring safety protection while achieving smooth operation of the AGV.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a multi-sensor fusion outdoor AGV extreme weather safety protection method, comprising the following steps:

[0006] 1) Synchronize the time of the laser radar, infrared thermal imaging camera, and 4D millimeter wave radar sensor installed on the AGV;

[0007] 2) Calibrate the internal and external parameters of each sensor to ensure spatial synchronization of laser radar, infrared thermal imaging, and 4D millimeter wave radar sensors;

[0008] 3) After collecting the raw data of each sensor, the sensor status abnormality is judged and the data abnormality is judged, and the abnormal data is processed to avoid subsequent calculation errors causing abnormal calculation of the protection zone;

[0009] 4) When the AGV is performing tasks and walking on the route, the operating status of the surrounding AGVs is obtained in real time through the dispatching system; combined with the AGV operating status, the data of each sensor is fused at the feature level and decision level to obtain the safety protection area or obstacle information;

[0010] 5) The rationality of the protection area is judged. If the protection area meets the rationality, the AGV drivable area is further calculated based on the obtained safety protection area, and the AGV driving route and speed planning in the safety protection area are planned to control the AGV operation; otherwise, if it does not meet the rationality, the protection area is determined to be smaller than the required size, and the AGV is controlled to stop for protection.

[0011] The step 1) is specifically:

[0012] The time synchronization of the laser radar, infrared thermal imaging camera, and 4D millimeter wave radar sensor on the AGV is achieved through the Client / Server timing method of NPT. Each sensor is used as the Client, and the main controller inside the AGV is used as the Server. Each sensor communicates with the main controller to obtain the delay of the request packet and the reply packet, thereby achieving the purpose of time synchronization between each sensor and the main controller.

[0013] In step 2), the laser radar is used as the main sensor. First, a main laser radar is calibrated, and then other radars are calibrated to the main laser radar in turn, and finally all sensors are calibrated to the AGV coordinate system.

[0014] Calibrate the internal and external parameters of the laser radar, specifically:

[0015] When calibrating the laser radar, a plane perpendicular to the ground is selected as the calibration plane. The plane is fitted by the laser point cloud data, and the normal vector is calculated to calibrate the pitch angle, roll angle, and z-direction offset, that is:

[0016] The fitting plane of laser point cloud data is:

[0017] Ax+By+Cz+d=0

[0018] Among them, A, B, C, and d are the plane equation parameters, and (x, y, z) are the point cloud coordinates;

[0019] 1-1) Get the pitch angle:

[0020]

[0021] Among them, pitch i is the pitch angle calculated in a single time, and pitch is the average pitch angle;

[0022] 1-2) Get the roll angle:

[0023]

[0024] Among them, roll i is the roll angle calculated in a single time, and roll is the average roll angle;

[0025] 1-3) Get the laser radar height:

[0026]

[0027] Among them, z i is the height of a single calculation, and z is the average height.

[0028] The infrared thermal imaging camera is calibrated with internal and external parameters, specifically:

[0029] For infrared thermal imaging cameras, a calibration method based on reference objects is used. Through reference objects of known size and shape, feature points are extracted from 2D images, and the corresponding positions of feature points are found in the lidar point cloud. The PnP algorithm is used to calculate the conversion matrix between sensors, and then the camera's posture is solved, that is:

[0030] 2-1) The homogeneous coordinates of the two coordinate points in the infrared camera pixel coordinate system (u, v) and the lidar world coordinate system (X, Y, Z) satisfy the following formula:

[0031]

[0032] 2-2) After sorting, we get the following two equations, where [P i ] represents the projection matrix P 3×4 The i-th row of

[0033]

[0034] 2-3) Consider the matrix P 3×4 There are 12 unknowns in total, so there must be at least 6 sets of corresponding points, that is, N ≥ 6, and the corresponding matrix is ​​expressed as:

[0035]

[0036] 2-4) In summary, remember

[0037] Among them, the i-th column of P3×4 is (P i);

[0038] 2-5) Solve P by convex optimization method 3×4 There are 12 unknowns in the projection matrix. Since the first three columns of the projection matrix are essentially rotation matrices, the constraint condition is the orthogonality of the rotation matrix, that is:

[0039] minimize∑((u i -k1) 2 +(v i -k2) 2 )

[0040]

[0041] The internal and external parameters of the 4D millimeter wave radar sensor are calibrated as follows:

[0042] The 4D millimeter-wave radar adopts a manual calibration method based on the laser radar data. The 4D millimeter-wave radar data and the data collected by the laser radar are displayed in the same window, and the installation position parameters of the 4D millimeter-wave radar are manually adjusted. When the 4D millimeter-wave radar data and the laser point cloud data are aligned, the calibration is completed.

[0043] The AGV operation status includes: the position and speed, acceleration, and local environment model of the AGV during operation.

[0044] In step 4), the data of each sensor is fused at the feature level and decision level in combination with the AGV operation status, specifically:

[0045] 3-1) The data processing of the infrared thermal imaging camera is performed by extracting features through a thermal infrared image enhancement algorithm based on deep learning;

[0046] 3-2) 4D millimeter-wave radar uses the DBSCAN method to cluster point cloud data, distinguish potential targets, and perform feature extraction. The features obtained by the millimeter-wave radar and the distance and speed information contained are integrated with the feature information of the infrared thermal imaging camera to generate high-level environmental perception features;

[0047] 3-3) The laser radar mainly extracts features through filtering and point cloud segmentation, and associates the extracted features with the features extracted from the infrared thermal imaging camera and 4D millimeter wave radar for data fusion, namely:

[0048] Considering decision fusion, since each sensor performs feature extraction and classification decisions based on its own measurement results;

[0049] Therefore, the probability distribution value is calculated for the decision result of each sensor, and then fused using the Dempster algorithm, that is:

[0050] Assume that two sensors i and j are fused:

[0051] D. Generate the basic probability decision of the sensor: the support degree of sensor i for proposition H is m i (H), the support degree of sensor j for proposition H is m j (H);

[0052] E. Calculate the conflict coefficient of different sensors: This shows that B and C are two conflicting propositions;

[0053] F. Synthetic Evidence:

[0054] Assuming there are multiple sensors, it is necessary to continuously fuse the results of two sensors with the results of subsequent sensors;

[0055] 3-4) Use different data fusion methods for different areas in the predicted driving area.

[0056] In step 3-4), different data fusion methods are used for different areas in the predicted driving area, specifically:

[0057] 4-1) Combine the AGV body information and the dispatching system information to segment the predicted AGV drivable area. Different data fusion methods are used for different areas. The closer the area is to the AGV, the higher the safety protection requirements are, and the more complex the fusion algorithm is.

[0058] 4-2) When the surrounding targets identified by each sensor are basically the same, it is determined that the weather is relatively good, and the raw data of the direct sensors are fused for safety protection;

[0059] 4-3) When there are large differences in the targets recognized by each sensor, it is determined that abnormal weather may exist, and image enhancement and feature complementarity-based strategies are used to fuse the laser radar, infrared thermal imaging camera, and 4D millimeter wave radar sensor on the AGV. Historical data is taken into account during the data fusion process to avoid interference from abnormal data.

[0060] In step 5), the AGV drivable area is further calculated based on the obtained safety protection area, and the AGV driving route and speed planning in the safety protection area are planned to control the operation of the AGV, specifically:

[0061] 5-1) Generate a local map based on the detected target information, which includes obstacle information;

[0062] 5-2) Determine the positional relationship between the current AGV driving path and the obstacles in the map. If there is interference, the AGV will be controlled to stop for protection or re-plan the path;

[0063] (1) Parking protection mainly uses the distance s between the AGV and the obstacle to control the speed planning of the vehicle. During the planning process, the variable acceleration a is used for planning control. Specifically:

[0064]

[0065] v=v0-a*t

[0066] Among them, v0 represents the initial speed, v represents the control speed;

[0067] (2) For re-planning the path, a new planning path is generated through the Hybrid A* algorithm.

[0068] The present invention has the following beneficial effects and advantages:

[0069] 1. The safety protection of outdoor automatic guided vehicles (AGVs) of the present invention is a key issue because they need to ensure the safety of personnel, equipment and goods while operating in complex, changeable and unpredictable environments such as all-weather and extreme weather.

[0070] 2. The method of the present invention can significantly improve the safety performance of outdoor AGVs under extreme weather conditions, ensure that AGVs can operate safely, reliably, efficiently and stably in complex and changeable outdoor environments, and achieve the purpose of human-machine collaborative work.

[0071] 3. The present invention provides an innovative outdoor AGV safety protection method. The reasons for false alarms and missed alarms in the safety protection of outdoor AGVs in extreme weather are analyzed. Based on this, a multi-terminal data fusion technology using laser radar, infrared thermal imaging camera, and 4D millimeter-wave radar is proposed to adapt to the safety protection method of outdoor AGVs in extreme weather. The method greatly improves the robustness and accuracy of the safety protection of outdoor AGVs in abnormal weather (rain, snow, fog, etc.), and ensures the smooth operation of AGVs while ensuring safety protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A method flow chart of the safety protection method for outdoor AGV under extreme weather conditions of the present invention;

[0073] Figure 2 Schematic diagram of the multi-sensor data fusion of the present invention. DETAILED DESCRIPTION

[0074] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0075] In order to achieve safety protection for AGVs in extreme weather conditions, this method uses three sensors: lidar, infrared thermal imaging camera and 4D millimeter-wave radar. When the effective measurement range of the lidar is reduced due to abnormal weather, the advantages of red infrared thermal imaging and 4D millimeter-wave radar become obvious. Compared with lidar, millimeter-wave radar has a longer wavelength, and 4D millimeter-wave radar is less affected by weather. Environmental dirt or water droplet refraction caused by severe weather conditions will not affect its work; at the same time, 4D millimeter-wave radar can penetrate obstacles in front and detect obstacles in front that the lidar cannot detect; infrared thermal imaging cameras have a stronger ability to penetrate fog, haze, rain and snow, so the imaging effect is almost unaffected under severe weather conditions.

[0076] Accurate multi-sensor time and space unification is a prerequisite for multi-sensor fusion perception. The safety protection plan under extreme weather conditions must first achieve multi-sensor data time and space synchronization, raw data collection and preliminary processing; combined with the AGV surrounding environment model, considering the AGV body information, scheduling system information and historical sensor data, strengthen the data fusion between sensors, and due to the reduction of sensor performance in extreme weather, sensor data enhancement and feature difference compensation algorithms are performed. The weighted average method is used to increase the weight of infrared thermal imaging camera and 4D millimeter wave radar sensor data fusion, thereby improving the reliability and stability of AGV safety protection under extreme weather conditions. The robustness of the overall perception system is improved by complementing each other's advantages.

[0077] In order to adapt to extreme weather, sensor data processing uses algorithms such as denoising, filtering, feature extraction, and data fusion. In order to reduce computing power, data processing and fusion are divided into different regions for processing and fusion. Different regions use different processing and fusion strategies. The regions are mainly divided into three areas: far, medium, and near. The far-distance area is directly processed and calculated on the original data, which can reduce computing power in large scenarios. The medium-distance area is mainly fused based on target features. Features are first extracted from sensor data, and then fused at the feature level. The short-distance area is the most critical area for AGV safety protection. AGV decisions are mainly added on the basis of sensor data feature fusion to achieve a composite fusion of features and decisions.

[0078] Based on the above, the safe driving range of AGV is obtained, which is used to guide the AGV decision-making system to make correct decisions on the operation of AGV and ensure the safe and reliable operation of AGV.

[0079] like Figure 1 As shown, it is a method flow chart of the safety protection method of outdoor AGV under extreme weather conditions of the present invention. The safety protection method of outdoor AGV under extreme weather conditions with multi-sensor fusion of the present invention comprises the following steps:

[0080] 1) The time synchronization of the laser radar, infrared thermal imaging camera, and 4D millimeter wave radar sensor on the AGV is achieved through the Client / Server timing method of NPT, that is, each sensor is used as the client, and the main controller inside the AGV is used as the server. Each sensor communicates with the main controller to obtain the delay of the request packet and the reply packet, thereby achieving the purpose of time synchronization between each sensor and the main controller;

[0081] 2) Calibrate the internal and external parameters of each sensor. Take the laser radar as the main sensor, calibrate the main laser radar first, and then calibrate the other radars to the main laser radar in turn, and finally calibrate all sensors to the AGV coordinate system to ensure that the laser radar, infrared thermal imaging, and 4D millimeter wave radar sensor achieve spatial synchronization;

[0082] Calibrate the internal and external parameters of the laser radar, specifically:

[0083] When calibrating the laser radar, a plane perpendicular to the ground is selected as the calibration plane. The plane is fitted by the laser point cloud data, and the normal vector is calculated to calibrate the pitch angle, roll angle, and z-direction offset, that is:

[0084] The fitting plane of laser point cloud data is:

[0085] Ax+By+Cz+d=0

[0086] Among them, A, B, C, and d are the plane equation parameters, and (x, y, z) are the point cloud coordinates;

[0087] 1-1) Get the pitch angle:

[0088]

[0089] Among them, pitch i is the pitch angle calculated in a single time, and pitch is the average pitch angle;

[0090] 1-2) Get the roll angle:

[0091]

[0092] Among them, roll i is the roll angle calculated in a single time, and roll is the average roll angle;

[0093] 1-3) Get the laser radar height:

[0094]

[0095] Among them, z i is the height of a single calculation, and z is the average height.

[0096] The infrared thermal imaging camera is calibrated with internal and external parameters, specifically:

[0097] For infrared thermal imaging cameras, a calibration method based on reference objects is used. Through reference objects of known size and shape, feature points are extracted from 2D images, and the corresponding positions of feature points are found in the lidar point cloud. The PnP algorithm is used to calculate the conversion matrix between sensors, and then the camera's posture is solved, that is:

[0098] 2-1) The homogeneous coordinates of the two coordinate points in the infrared camera pixel coordinate system (u, v) and the lidar world coordinate system (X, Y, Z) satisfy the following formula:

[0099]

[0100] 2-2) After sorting, we get the following two equations, where [P i ] represents the projection matrix P 3×4 The i-th row of

[0101]

[0102] 2-3) Consider the matrix P 3×4 There are 12 unknowns in total, so there must be at least 6 sets of corresponding points, that is, N ≥ 6, and the corresponding matrix is ​​expressed as:

[0103]

[0104] 2-4) In summary, remember

[0105] Among them, the i-th column of P3×4 is (P i );

[0106] 2-5) Solve P by convex optimization method 3×4 There are 12 unknowns in the projection matrix. Since the first three columns of the projection matrix are essentially rotation matrices, the constraint condition is the orthogonality of the rotation matrix, that is:

[0107] minimize∑((u i -k1) 2 +(v i -k2) 2 )

[0108]

[0109] The internal and external parameters of the 4D millimeter wave radar sensor are calibrated as follows:

[0110] The 4D millimeter-wave radar adopts a manual calibration method based on the laser radar data. The 4D millimeter-wave radar data and the data collected by the laser radar are displayed in the same window, and the installation position parameters of the 4D millimeter-wave radar are manually adjusted. When the 4D millimeter-wave radar data and the laser point cloud data are aligned, the calibration is completed.

[0111] 3) After collecting the raw data of each sensor, the sensor status abnormality is judged and the data abnormality is judged, and the abnormal data is processed to avoid subsequent calculation errors causing abnormal calculation of the protection zone;

[0112] 4) When the AGV is performing tasks and walking routes, the operating status of the surrounding AGVs can be obtained in real time through the scheduling system;

[0113] The AGV operation status includes: the position and speed, acceleration, and local environment model of the AGV during operation.

[0114] like Figure 2 As shown, it is a principle diagram of multi-sensor data fusion of the present invention. The present invention combines the AGV running status, fuses the data of each sensor at the feature level and the decision level, and obtains the safety protection area or obstacle information;

[0115] 4-1) The infrared thermal imaging camera data processing performs feature extraction through a thermal infrared image enhancement algorithm based on deep learning;

[0116] 4-2) 4D millimeter-wave radar uses the DBSCAN method to cluster point cloud data, distinguish potential targets, and perform feature extraction. The features obtained by the millimeter-wave radar and the distance and speed information contained are integrated with the feature information of the infrared thermal imaging camera to generate high-level environmental perception features;

[0117] 4-3) The laser radar mainly extracts features through filtering and point cloud segmentation, and associates the extracted features with the features extracted from the infrared thermal imaging camera and 4D millimeter wave radar for data fusion, namely:

[0118] Considering decision fusion, since each sensor performs feature extraction and classification decisions based on its own measurement results;

[0119] Therefore, the probability distribution value is calculated for the decision result of each sensor, and then fused using the Dempster algorithm, that is:

[0120] Assume that two sensors i and j are fused:

[0121] G. Generate the basic probability decision of the sensor: the support degree of sensor i for proposition H is m i(H), the support degree of sensor j for proposition H is m j (H);

[0122] H. Calculate the conflict coefficient of different sensors: This shows that B and C are two conflicting propositions;

[0123] I. Synthetic Evidence:

[0124] Assuming there are multiple sensors, it is necessary to continuously fuse the results of two sensors with the results of subsequent sensors;

[0125] 4-4) Use different data fusion methods for different areas in the predicted driving area.

[0126] Furthermore, different data fusion methods are used for different areas in the predicted driving area, specifically:

[0127] a. Combine the AGV body information and the dispatching system information to segment the predicted AGV drivable area. Different data fusion methods are used for different areas. The closer the area is to the AGV, the higher the safety protection requirements are, and the more complex the fusion algorithm is;

[0128] b. When the surrounding targets identified by each sensor are basically the same, it is determined that the weather is relatively good, and the original data of the direct sensor is fused for safety protection;

[0129] c. When there are large differences in the targets identified by each sensor, it is determined that abnormal weather may exist. Image enhancement and feature-complementary strategies are used to fuse the laser radar, infrared thermal imaging camera, and 4D millimeter-wave radar sensor on the AGV. Historical data is considered in the data fusion process to avoid interference from abnormal data.

[0130] 5) The rationality of the protection area is judged. If the protection area meets the rationality, the AGV drivable area is further calculated based on the obtained safety protection area, and the AGV driving route and speed planning in the safety protection area are planned to control the AGV operation; otherwise, if it does not meet the rationality, the protection area is determined to be smaller than the required size, and the AGV is controlled to stop for protection.

[0131] Among them, according to the obtained safety protection area, the AGV drivable area is further calculated, and the AGV driving route and speed planning in the safety protection area are planned to control the AGV operation, specifically:

[0132] 5-1) Generate a local map based on the detected target information, which includes obstacle information;

[0133] 5-2) Determine the positional relationship between the current AGV driving path and the obstacles in the map. If there is interference, the AGV will be controlled to stop for protection or re-plan the path;

[0134] (1) Parking protection mainly uses the distance s between the AGV and the obstacle to control the speed planning of the vehicle. During the planning process, the variable acceleration a is used for planning control. Specifically:

[0135]

[0136] v=v0-a*t

[0137] Among them, v0 represents the initial speed, v represents the control speed;

[0138] (2) For re-planning the path, a new planning path is generated through the Hybrid A* algorithm.

[0139] Embodiment 1:

[0140] The specific implementation mainly includes the following main stages:

[0141] Phase 1: Sensor temporal and spatial synchronization and data acquisition

[0142] 1. Synchronization of time for laser radar, infrared thermal imaging camera, and 4D millimeter wave radar sensor. Each sensor has its own internal clock, but since the clock drift of each clock is different, there is a time difference between the clocks, so it is necessary to unify the time of each sensor. This solution mainly uses the Client / Server timing method of NPT (Network Time Protocol). Each sensor is equivalent to the client, and the main controller inside the AGV is equivalent to the server. Each sensor communicates with the main controller to obtain the delay of the request packet and the reply packet, thereby achieving the purpose of time synchronization between each sensor and the main controller.

[0143] 2. Spatial synchronization of laser radar, infrared thermal imaging, and 4D millimeter wave radar sensors. Calibrate the internal and external parameters of each sensor to ensure spatial synchronization. In this method, laser radar is used as the main sensor. First, a main laser radar is calibrated, and then other radars are calibrated to the main laser radar in turn, and finally all sensors are calibrated to the AGV coordinate system.

[0144] 3. Single LiDAR calibration requires a large plane as the ground. The plane is fitted with laser point cloud data and the normal vector is calculated, so that the pitch, roll, and z-direction offset can be calibrated. The lateral (y), longitudinal (x) offset, and raw (yaw angle) are mainly obtained by mechanical measurement and manual measurement compensation.

[0145] 4. For infrared thermal imaging cameras, a calibration method based on reference objects is used. Reference objects of known size and shape (such as a chessboard) are used to extract feature points from 2D images, and the corresponding positions of these feature points are found in the lidar point cloud. The PnP (Perspective-n-Point) algorithm is used to calculate the transformation matrix between sensors, and then the camera's posture is solved.

[0146] 5.4D millimeter wave radar adopts a manual calibration method based on laser radar data. The 4D millimeter wave radar data and the data collected by the laser radar are displayed in the same window. The installation position and other parameters of the 4D millimeter wave radar are manually adjusted. When the 4D millimeter wave radar data and the laser point cloud data are aligned, the calibration is completed.

[0147] 6. Collect raw data from each sensor. During this process, it is necessary to judge the abnormal state of the sensor (such as sensor disconnection, etc.) and judge the data abnormality to avoid subsequent calculation errors that cause abnormal calculation of the protection zone.

[0148] Phase 2: Considering the fusion of multi-terminal data and extreme weather data

[0149] 1. Comprehensively consider the multi-terminal data. The tasks performed by AGV and the routes it travels are controlled by the scheduling system. The operation status of surrounding vehicles can be obtained from the scheduling system at any time. The position, speed, acceleration, and local environmental model of the AGV during operation are known. Based on these conditions, the driving area occupied by the AGV in the future can be roughly predicted. In order to reduce the computing power of data fusion, the predicted driving area is obtained by combining the above information, and data fusion is only performed on this area.

[0150] 2. LiDAR, infrared thermal imaging cameras, and 4D millimeter-wave radar sensors each have their own advantages and disadvantages. Especially in abnormal weather conditions, in order to achieve safe and reliable protection, it is necessary to give full play to the advantages of each sensor and make up for its disadvantages. In abnormal weather conditions, infrared thermal imaging cameras and 4D millimeter-wave radars are less affected by abnormal weather, but the data from infrared thermal imaging cameras is 2D image data and lacks depth data support; 4D millimeter-wave radar data carries distance (depth) information, so the fusion of the two data can improve the perception of the surrounding environment, and at the same time combine the LiDAR sensor data to achieve safety protection in extreme weather conditions.

[0151] 3. The data processing of infrared thermal imaging cameras is mainly based on the thermal infrared image enhancement algorithm of deep learning, which can improve image quality, remove noise, enhance details, and extract features through deep learning; 4D millimeter wave radar uses DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method to cluster point cloud data, distinguish potential targets, and extract features. The features obtained by the millimeter wave radar and the distance and speed information contained are integrated with the feature information of the infrared thermal imaging camera to generate high-level environmental perception features;

[0152] 4. LiDAR mainly extracts features through filtering, point cloud segmentation, etc., and then associates data with the features in the previous step, and then performs data fusion. In this process, decision fusion is considered. Each sensor extracts features and makes classification decisions based on its own measurement results. After that, the probability distribution value is calculated for the decision results of each sensor, and then the Dempster algorithm is used for fusion.

[0153] 5. To sum up the above, combined with the AGV body information and the dispatching system information, the predicted driving area is segmented, and different data fusion methods are used in different areas. The closer the area is to the AGV, the higher the safety protection requirements, and the higher the complexity of the fusion algorithm used. When the surrounding targets identified by each sensor are basically the same, it is considered to be relatively good weather, and the original data of the sensor can be directly fused for safety protection. When there are large differences in the targets identified by each sensor, there may be abnormal weather. Image enhancement and feature-complementary-based strategies are used for sensor data fusion. Historical data is considered in the process of data fusion to avoid interference from abnormal data and improve the robustness and accuracy of the fusion process.

[0154] Phase 3: AGV decision control

[0155] 1. Further calculate the AGV drivable area based on the safety protection area obtained in the second stage to realize the AGV operation route planning in the drivable area;

[0156] 2. Control the running speed of AGV to ensure that it can stop in time when there are potential safety hazards in the safety protection area to ensure the safety of people, vehicles and objects.

[0157] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-sensor fusion outdoor AGV extreme weather safety protection method, characterized in that: The following steps are involved: 1) Synchronize the time of the laser radar, infrared thermal imaging camera, and 4D millimeter wave radar sensor installed on the AGV; 2) Calibrate the internal and external parameters of each sensor to ensure spatial synchronization of laser radar, infrared thermal imaging, and 4D millimeter wave radar sensors; 3) After collecting the raw data of each sensor, the sensor status abnormality is judged and the data abnormality is judged, and the abnormal data is processed to avoid subsequent calculation errors causing abnormal calculation of the protection zone; 4) When the AGV is performing tasks and walking on the route, the operating status of the surrounding AGVs is obtained in real time through the dispatching system; combined with the AGV operating status, the data of each sensor is fused at the feature level and decision level to obtain the safety protection area or obstacle information; 5) The rationality of the protection area is judged. If the protection area meets the rationality, the AGV drivable area is further calculated based on the obtained safety protection area, and the AGV driving route and speed planning in the safety protection area are planned to control the AGV operation; otherwise, if it does not meet the rationality, the protection area is determined to be smaller than the required size, and the AGV is controlled to stop for protection.

2. According to the multi-sensor fusion outdoor AGV extreme weather safety protection method of claim 1, it is characterized in that: The step 1) is specifically: The time synchronization of the laser radar, infrared thermal imaging camera, and 4D millimeter wave radar sensor on the AGV is achieved through the Client / Server timing method of NPT. Each sensor is used as the Client, and the main controller inside the AGV is used as the Server. Each sensor communicates with the main controller to obtain the delay of the request packet and the reply packet, thereby achieving the purpose of time synchronization between each sensor and the main controller.

3. The multi-sensor fusion outdoor AGV extreme weather safety protection method according to claim 1 is characterized in that: In step 2), the laser radar is used as the main sensor. First, a main laser radar is calibrated, and then other radars are calibrated to the main laser radar in turn, and finally all sensors are calibrated to the AGV coordinate system.

4. The multi-sensor fusion outdoor AGV extreme weather safety protection method according to claim 1 is characterized in that: Calibrate the internal and external parameters of the laser radar, specifically: When calibrating the laser radar, a plane perpendicular to the ground is selected as the calibration plane. The plane is fitted by the laser point cloud data, and the normal vector is calculated to calibrate the pitch angle, roll angle, and z-direction offset, that is: The fitting plane of laser point cloud data is: Ax+By+Cz+d=0 Among them, A, B, C, and d are the plane equation parameters, and (x, y, z) are the point cloud coordinates; 1-1) Get the pitch angle: Among them, pitch i is the pitch angle calculated in a single time, and pitch is the average pitch angle; 1-2) Get the roll angle: Among them, roll i is the roll angle calculated in a single time, and roll is the average roll angle; 1-3) Get the laser radar height: Among them, z i is the height of a single calculation, and z is the average height.

5. The multi-sensor fusion outdoor AGV extreme weather safety protection method according to claim 1 is characterized in that: The infrared thermal imaging camera is calibrated with internal and external parameters, specifically: For infrared thermal imaging cameras, a calibration method based on reference objects is used. Through reference objects of known size and shape, feature points are extracted from 2D images, and the corresponding positions of feature points are found in the lidar point cloud. The PnP algorithm is used to calculate the conversion matrix between sensors, and then the camera's posture is solved, that is: 2-1) The homogeneous coordinates of the two coordinate points in the infrared camera pixel coordinate system (u, v) and the lidar world coordinate system (X, Y, Z) satisfy the following formula: 2-2) After sorting, we get the following two equations, where [P i ] represents the projection matrix P 3×4 The i-th row of 2-3) Consider the matrix P 3×4 There are 12 unknowns in total, so there must be at least 6 sets of corresponding points, that is, N ≥ 6, and the corresponding matrix is ​​expressed as: 2-4) In summary, remember Among them, P 3×4 The i-th column of i ); 2-5) Solve P by convex optimization method 3×4 There are 12 unknowns in the projection matrix. Since the first three columns of the projection matrix are essentially rotation matrices, the constraint condition is the orthogonality of the rotation matrix, that is: minimize∑((u i -k1) 2 +(v i -k2) 2 ) 6. The multi-sensor fusion outdoor AGV extreme weather safety protection method according to claim 1 is characterized in that: The internal and external parameters of the 4D millimeter wave radar sensor are calibrated as follows: The 4D millimeter-wave radar adopts a manual calibration method based on the laser radar data. The 4D millimeter-wave radar data and the data collected by the laser radar are displayed in the same window, and the installation position parameters of the 4D millimeter-wave radar are manually adjusted. When the 4D millimeter-wave radar data and the laser point cloud data are aligned, the calibration is completed.

7. The multi-sensor fusion outdoor AGV extreme weather safety protection method according to claim 1 is characterized in that: The AGV operation status includes: the position and speed, acceleration, and local environment model of the AGV during operation.

8. The multi-sensor fusion outdoor AGV extreme weather safety protection method according to claim 1 is characterized in that: In step 4), the data of each sensor is fused at the feature level and decision level in combination with the AGV operation status, specifically: 3-1) The data processing of the infrared thermal imaging camera is performed by extracting features through a thermal infrared image enhancement algorithm based on deep learning; 3-2) 4D millimeter-wave radar uses the DBSCAN method to cluster point cloud data, distinguish potential targets, and perform feature extraction. The features obtained by the millimeter-wave radar and the distance and speed information contained are integrated with the feature information of the infrared thermal imaging camera to generate high-level environmental perception features; 3-3) The laser radar mainly extracts features through filtering and point cloud segmentation, and associates the extracted features with the features extracted from the infrared thermal imaging camera and 4D millimeter wave radar for data fusion, namely: Considering decision fusion, since each sensor performs feature extraction and classification decisions based on its own measurement results; Therefore, the probability distribution value is calculated for the decision result of each sensor, and then fused using the Dempster algorithm, that is: Assume that two sensors i and j are fused: A. Generate the basic probability decision of the sensor: the support degree of sensor i for proposition H is m i (H), the support degree of sensor j for proposition H is m j (H); B. Calculate the conflict coefficient of different sensors: This shows that B and C are two conflicting propositions; C. Synthetic evidence: Assuming there are multiple sensors, it is necessary to continuously fuse the results of two sensors with the results of subsequent sensors; 3-4) Use different data fusion methods for different areas in the predicted driving area.

9. The multi-sensor fusion outdoor AGV extreme weather safety protection method according to claim 8 is characterized in that: In step 3-4), different data fusion methods are used for different areas in the predicted driving area, specifically: 4-1) Combine the AGV body information and the dispatching system information to segment the predicted AGV drivable area. Different data fusion methods are used for different areas. The closer the area is to the AGV, the higher the safety protection requirements are, and the more complex the fusion algorithm is. 4-2) When the surrounding targets identified by each sensor are basically the same, it is determined that the weather is relatively good, and the raw data of the direct sensors are fused for safety protection; 4-3) When there are large differences in the targets recognized by each sensor, it is determined that abnormal weather may exist, and image enhancement and feature complementarity-based strategies are used to fuse the laser radar, infrared thermal imaging camera, and 4D millimeter wave radar sensor on the AGV. Historical data is taken into account during the data fusion process to avoid interference from abnormal data.

10. The multi-sensor fusion outdoor AGV extreme weather safety protection method according to claim 1 is characterized in that: In step 5), the AGV drivable area is further calculated based on the obtained safety protection area, and the AGV driving route and speed planning in the safety protection area are planned to control the operation of the AGV, specifically: 5-1) Generate a local map based on the detected target information, which includes obstacle information; 5-2) Determine the positional relationship between the current AGV driving path and the obstacles in the map. If there is interference, the AGV will be controlled to stop for protection or re-plan the path; (1) Parking protection mainly uses the distance s between the AGV and the obstacle to control the speed planning of the vehicle. During the planning process, the variable acceleration a is used for planning control. Specifically: v=v0-a*t Among them, v0 represents the initial speed, v represents the control speed; (2) For re-planning the path, a new planning path is generated through the Hybrid A* algorithm.

Images