Unmanned vehicle path planning method and system based on cooperation of unmanned aerial vehicle

Abstract

The invention discloses an unmanned vehicle path planning method and system based on unmanned aerial vehicle cooperation. The method comprises the following steps: firstly, carrying out environmental data acquisition through a laser radar and a camera carried by an unmanned aerial vehicle, realizing space-time calibration of sensor data, constructing a high-precision three-dimensional point cloud map by using an SLAM technology, and carrying out pose map optimization through a GTSAM optimizer; secondly, performing semantic segmentation on an image shot by the unmanned aerial vehicle by adopting a deep learning technology, identifying a drivable area, and associating the drivable area with the point cloud data to form a semantic point cloud map; and on the basis, by calculating the width, gradient and continuity of the road, a path suitable for the unmanned vehicle to run is screened out, and path planning is carried out. According to the invention, the path planning capability of the unmanned vehicle in a complex environment is effectively improved, and powerful technical support is provided for disaster rescue and material distribution.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned vehicles, and in particular to a method and system for unmanned vehicle path planning in cooperation with a drone. Background Art

[0002] With the rapid development of science and technology, autonomous driving technology has become an important research direction in the field of modern transportation. The core of autonomous driving technology is to achieve autonomous navigation, decision-making and control of vehicles to improve transportation efficiency, reduce accident rates, and provide possibilities for material distribution in special environments. In this field, the collaborative operation of unmanned vehicles and drones is particularly eye-catching. Through efficient information exchange and task allocation, they can significantly improve the efficiency and balance of material distribution.

[0003] The existing solutions for collaborative path planning between vehicles and drones mainly focus on how to achieve efficiency and balance in material distribution through collaborative operations between the two. For example, according to order requirements, the materials to be delivered are divided into two categories: drone delivery and vehicle delivery, and the starting and ending points of the delivery are clearly defined. Next, communication is established between drones and vehicles to ensure that they can alternately deliver materials. This alternating delivery method can ensure the balance of the entire material distribution, and also enable drones and vehicles to complement each other and give full play to their respective advantages. In terms of path planning, existing solutions usually use path planning algorithms such as the A* algorithm or the Dijkstra algorithm to calculate the optimal path for drones and vehicles, and perform scheduling and execution. These algorithms can take into account a variety of factors, such as distance, time, road conditions, etc., to ensure that materials can be delivered along the optimal path.

[0004] However, although the existing solution considers the efficiency and balance of material distribution through the collaborative operation of drones and vehicles, this solution does not consider special scenarios, such as special disaster scenarios such as floods, earthquakes, mudslides, fires, etc., nor does it assign clear work responsibilities between drones and vehicles. Therefore, when encountering extreme scenarios, the existing technical solutions cannot meet the needs. Summary of the invention

[0005] Based on this, the embodiment of the present application provides a method and system for unmanned vehicle path planning in collaboration with drones, which is applied to scenes that are prone to road obstruction, such as floods, earthquakes, mudslides, and fires. The present application uses an electric multi-rotor drone to observe the environment in front of the unmanned vehicle in advance, collects point clouds and images of the scene through laser radar and cameras, and sends the fused information to the vehicle, intelligently selects drivable roads, and ensures that the vehicle plans its path within the drivable area. In this way, search and rescue and material supply vehicles can quickly arrive at the scene when man-made or natural disasters occur.

[0006] In a first aspect, a method for unmanned vehicle path planning in collaboration with a drone is provided, the method comprising:

[0007] S100, UAV sensor spatiotemporal calibration: synchronize the time of the UAV and the unmanned vehicle sensors, and calibrate the external parameters between the lidar, camera, and inertial navigation;

[0008] S200, SLAM builds point cloud map: uses SLAM algorithm to perform self-positioning and build 3D point cloud map, and stores point cloud data in PCD format;

[0009] S300, Image Semantic Segmentation: Apply deep learning algorithms to classify and identify image pixels in images collected by drones;

[0010] S400, image semantics and point cloud map association: convert the point cloud map from the global coordinate system to the lidar coordinate system, and use the external parameters obtained by calibration to convert the point cloud data to the camera coordinate system; through the camera projection model, each point in the point cloud is projected onto the corresponding pixel of the image;

[0011] S500, screening drivable roads: screening drivable roads in the projected point cloud map;

[0012] S600, path planning: determining the shortest path based on the screened drivable roads as the target planning path.

[0013] Optionally, S100 synchronizes the time of the sensors of the drone and the unmanned vehicle, including:

[0014] Use RTK's PPS+GPRMC to synchronize time with the drone domain controller. At the same time, use PTP to eliminate the time delay caused by network transmission and accurately obtain the timestamps of all sensors.

[0015] Optionally, the external parameters between the laser radar, camera, and inertial navigation are calibrated in S100, including:

[0016] Pre-calibrate the external parameters between the camera and the lidar, and between the lidar and the inertial navigation system;

[0017] When the drone is stationary, the checkerboard calibration plate is moved multiple times. By simultaneously capturing the images of the calibration plate in the lidar and camera, the spatial transformation relationship between the two is calculated, that is, the external parameters between the lidar and the camera.

[0018] When the UAV is moving, the recursive trajectories of the inertial navigation and lidar are calculated respectively, and the external parameters between the lidar and inertial navigation are calculated through the hand-eye calibration principle.

[0019] Optionally, S200 uses a SLAM algorithm to perform self-positioning and construct a three-dimensional point cloud map, including:

[0020] The GTSAM optimizer is used to optimize the pose graph, and the optimization objective function is:

[0021]

[0022] Where X is the desired lidar keyframe pose, including rotation and translation, t i is the translation component in X, T ij , are the estimated and observed values ​​of the relative pose between the i-th frame and the j-th frame, respectively. I, L, Loop, and G are the key frames corresponding to the IMU pre-integration constraint, the lidar relative pose constraint, the closed-loop constraint, and the GPS constraint, respectively.

[0023] Optionally, S300 applies a deep learning algorithm to classify and identify image pixels of the image collected by the drone, including:

[0024] A fully convolutional network is used to classify each pixel in the image; wherein the classification categories include at least soil, buildings, vegetation, water, roads, and sidewalks;

[0025] And the edge detection method is used to identify the boundary lines in the image.

[0026] Optionally, edge detection is used to identify the local changes in pixel intensity in the image using Sobel, Canny, or Prwitt to identify the boundary line, specifically including:

[0027] Preprocess the image to enhance edge features and reduce noise interference;

[0028] Apply a specific convolution kernel or multi-stage detection process to calculate the gradient size and direction of each pixel in the image; the gradient information reflects the rapid changes in image brightness;

[0029] Refine the edges through the non-maximum suppression step to ensure that the edges are as thin as possible;

[0030] Extract clear edge contours from the gradient image to complete the edge recognition process.

[0031] Optionally, in S400, the point cloud data is converted into a camera coordinate system using the external parameters obtained by calibration; each point in the point cloud is projected onto a corresponding pixel of the image through a camera projection model, including:

[0032] The point cloud data is further converted from the laser radar coordinate system to the camera coordinate system using the external parameters obtained during the calibration process; wherein the external parameters include the rotation matrix and the translation vector;

[0033] A pinhole projection model of the camera is applied to project each point in the point cloud onto a corresponding pixel of the image; wherein the camera internal parameters of the pinhole projection model include focal length and optical center position.

[0034] Optionally, the conditions for screening the drivable roads in the projected point cloud map in S500 at least include:

[0035] Calculate the road width. The road width is greater than 1.5 times the width of the driverless car. The road width is calculated by gridding the road point cloud to generate a grid map. At the same time, a 3D polynomial curve is fitted to the road point cloud. A straight line is drawn perpendicular to the tangent of a point on the curve. The Euclidean distance between the straight line and the grid map compared to the two points is determined as the road width.

[0036] Flat road: The road point cloud is processed in segments, and the PCL point cloud library is used to fit the road point cloud to a plane. The distance from the point to the plane is less than the threshold, and the points greater than the threshold are clustered. The OBB algorithm in PCL is used to calculate the minimum bounding box size and mark it as an obstacle or pothole.

[0037] Road continuity: The values ​​of the road point cloud in the ground direction are continuous, and discontinuous areas are marked to indicate road interruption.

[0038] Optionally, determining the shortest path as the target planning path based on the screened drivable roads in S600 includes:

[0039] The path planning schemes adopted are A* global path planning and Bessel local path planning; among them, the A* algorithm can reduce the number of search nodes while ensuring the shortest path; the Bessel curve can ensure the curvature continuity of the local path planning curve, and plan the smoothest possible desired trajectory for the vehicle while ensuring the shortest path.

[0040] In the second aspect, a UAV-coordinated unmanned vehicle path planning system is provided, the system comprising:

[0041] The calibration module is used to synchronize the time of the sensors of the drone and the unmanned vehicle, and to calibrate the external parameters between the lidar, camera, and inertial navigation system;

[0042] Build a point cloud module, which is used to perform self-positioning and build a three-dimensional point cloud map through the SLAM algorithm, and store the point cloud data in PCD format;

[0043] The semantic segmentation module is used to classify and identify image pixels of drone-collected images using deep learning algorithms;

[0044] The association module is used to convert the point cloud map from the global coordinate system to the lidar coordinate system, and use the external parameters obtained by calibration to convert the point cloud data to the camera coordinate system; through the camera projection model, each point in the point cloud is projected to the corresponding pixel of the image;

[0045] A screening module, used to screen the drivable roads in the projected point cloud map;

[0046] The output module is used to determine the shortest path as the target planning path based on the screened drivable roads.

[0047] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0048] (1) Use drones to collect on-site information and send the calculated drivable roads to rescue vehicles, which can greatly shorten the time for rescue and material transportation.

[0049] (2) The A*&Bessel planning method can be used to smooth the vehicle route based on the shortest path, ensuring that the vehicle can travel on a route with continuous curvature. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0051] Figure 1 A flowchart of the steps of a method for unmanned vehicle path planning in collaboration with a drone provided in an embodiment of the present application;

[0052] Figure 2 A point cloud schematic map provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of semantic segmentation provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a semantic point cloud provided in an embodiment of the present application;

[0055] Figure 5 A projection diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] In the description of the present invention, the terms "comprises", "has" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units added based on further optimization schemes conceived by the present invention.

[0058] In this proposal, abbreviations and key terms are first defined:

[0059] SLAM (Simultaneous localization and mapping): simultaneous positioning and mapping. Unmanned vehicles, drones, robots, etc. locate their own positions while building maps in unknown environments.

[0060] PTP (Precision Time Protocol): Precision Time Protocol is a network protocol defined by the IEEE 1588 standard for implementing high-precision time synchronization in Ethernet.

[0061] PPS (Pulse Per Second): Generates a synchronous pulse signal with a time period of 1s based on UTC (Coordinated Universal Time), and the pulse width is usually between 5ms-100ms;

[0062] GPRMC (Global Positioning System Recommended Minimum data): is one of the NMEA 0183 messages, containing information such as longitude and latitude, date (year, month, day) and UTC time (accurate to seconds), and is output through the standard serial port.

[0063] The present invention provides a method for unmanned vehicle path planning in collaboration with a drone. This method is intended to be used when natural disasters such as floods, earthquakes, mudslides, etc. occur and search and rescue or the transportation of supplies to the disaster area is required. The laser radar and inertial navigation on the drone can be used to locate and map the unknown area to obtain a 3D point cloud map. At the same time, the image obtained by the drone camera is semantically segmented, and the information of the two is fused to map the 3D point cloud map onto the semantic image to obtain a 3D road point cloud. The 3D road point cloud is analyzed to determine the road openness and disconnection status, and road information such as slope, width, curvature, etc. is extracted. This message is sent to the ground autonomous driving vehicle to screen out drivable roads. The autonomous driving vehicle then plans the optimal route based on the positioning coordinates by combining global path planning with local path planning, thereby improving the efficiency of the unmanned vehicle entering the disaster area. To achieve the above purpose, if Figure 1 The present invention provides a method for unmanned vehicle path planning in cooperation with a drone, comprising the following steps:

[0064] S100, UAV sensor spatiotemporal calibration: synchronize the time of the UAV and the unmanned vehicle sensors, and calibrate the external parameters between the lidar, camera, and inertial navigation.

[0065] In this step, sensor time synchronization is achieved, and the external parameters between the lidar, camera, and inertial navigation are calibrated. Specifically, in order to accurately perform sensor data fusion, it is necessary to ensure the time synchronization and space synchronization between the lidar and camera, inertial navigation, and RTK. To ensure time synchronization, use RTK's PPS+GPRMC to time the drone domain controller, and at the same time, use PTP to eliminate the time delay caused by network transmission, and accurately obtain the timestamps of all sensors. To ensure spatial synchronization, pre-calibrate the external parameters between the camera and lidar, and the lidar and inertial navigation. When the drone is stationary, move the checkerboard calibration plate multiple times, and by simultaneously capturing the images of the calibration plate in the lidar and camera, calculate the spatial transformation relationship between the two, that is, the external parameters between the lidar and the camera. When the drone is moving, calculate the recursive trajectories of the inertial navigation and lidar respectively, and calculate the external parameters between the lidar and the inertial navigation through the hand-eye calibration principle.

[0066] S200, SLAM builds point cloud map: uses SLAM algorithm to perform self-positioning and build three-dimensional point cloud map, and stores point cloud data in PCD format.

[0067] In this step, the integration of inertial navigation, laser radar, and RTK is achieved for positioning and mapping. Specifically:

[0068] The drone acquires environmental data in real time by carrying a laser radar, and uses the SLAM algorithm to estimate its own position and construct a three-dimensional point cloud map of the environment. In order to obtain high-precision point clouds in the global coordinate system, the GTSAM optimizer is used to optimize the pose graph. The objective function includes IMU pre-integration constraints, laser radar odometer constraints, closed-loop constraints, and RTK position constraints. After obtaining the precise pose, the laser radar point cloud is transferred to the global coordinate system, and finally the point cloud data is stored in the PCD (Point Cloud Data) file format.

[0069] The optimized objective function is as follows:

[0070]

[0071] Where X is the desired lidar keyframe pose, including rotation and translation, t i is the translation component in X, T ij , are the estimated and observed values ​​of the relative pose between the i-th frame and the j-th frame, respectively. I, L, Loop, and G are the key frames corresponding to the IMU pre-integration constraint, the lidar relative pose constraint, the closed-loop constraint, and the GPS constraint, respectively. Figure 2 , a schematic map of the point cloud provided in an embodiment of the present application is given.

[0072] S300, Image Semantic Segmentation: Apply deep learning algorithms to classify and identify image pixels in images collected by drones.

[0073] In this step, deep learning is applied to classify and identify image pixels. Specifically, the present invention adopts deep learning to perform semantic segmentation. The model used is a fully convolutional network (FCN), which classifies each pixel in the image. The categories include: soil, buildings, vegetation, water, roads, sidewalks, and each pixel has a corresponding semantic label. In order to more accurately define the range of drivable roads, edge detection is also required. Edge detection methods (such as Sobel, Canny, Prwitt, etc.) can identify boundary lines in images or point clouds. These boundaries are often the dividing lines between roads and surrounding environments (such as buildings, mountains, rivers). By extracting these boundary lines, the outline of the drivable road can be further refined. Figure 3 A schematic diagram of semantic segmentation provided in an embodiment of the present application is given.

[0074] The edge detection method is used to identify the local changes in pixel intensity in the image using Sobel, Canny, and Prwitt to identify the boundary line, which specifically includes: preprocessing the image to enhance edge features and reduce noise interference; applying a specific convolution kernel or a multi-stage detection process to calculate the gradient size and direction of each pixel in the image; the gradient information reflects the rapid changes in image brightness; refining the edge through a non-maximum suppression step to ensure that the edge is as thin as possible; extracting a clear edge contour from the gradient map to complete the edge recognition process.

[0075] S400, image semantics is associated with the point cloud map: the point cloud map is converted from the global coordinate system to the lidar coordinate system, and the point cloud data is converted to the camera coordinate system using the external parameters obtained by calibration; each point in the point cloud is projected onto the corresponding pixel of the image through the camera projection model.

[0076] In this step, the road point cloud is extracted through coordinate system conversion and camera projection model. Specifically, in order to assign semantic information to the point cloud map, the point cloud is projected onto the image to obtain the semantic labels on the corresponding pixels. First, the point cloud map is converted from the global coordinate system to the lidar coordinate system using the positioning information. Then, the point cloud is converted from the lidar coordinate system to the camera coordinate system using the external parameters between the lidar and the camera. The camera projection model is then used to project the point cloud to a certain pixel in the image. In this process, point cloud statistical filtering, downsampling, regional growing and other operations are also required to ensure the correctness and continuity of the semantic labels. Figure 4 , a schematic diagram of the semantic point cloud provided in an embodiment of the present application is given.

[0077] like Figure 5 In the projection diagram shown, XL, YL, ZL define the laser radar coordinate system, Xc, Yc, Zc define the camera coordinate system, and point P is a point in the point cloud in the laser radar coordinate system. The point P(xl, yl, zl) is transferred from the laser radar coordinate system to the camera coordinate system as P'(xc, yc, zc), R ext ,t ext is the external parameter between the previously calibrated laser radar coordinates and the camera translation matrix, including the rotation matrix and the translation vector, then

[0078]

[0079] The position of P' in the camera coordinate system projected to the physical imaging plane through the pinhole model is p(u,v):

[0080]

[0081] Among them, f x , f y , c x, c y It is the focal length and optical center position, which are the factory parameters of the camera.

[0082] From the above, we can get the correspondence between the LiDAR point cloud and the camera pixels. From this, the point cloud obtains the semantic labels of the pixels to form a semantic point cloud, and then obtains the road point cloud.

[0083] S500, screening drivable roads: screening drivable roads in the projected point cloud map.

[0084] In this step, the road width and slope are calculated to determine whether the road surface is continuous and flat. Specifically, the present invention needs to meet the following conditions to screen drivable roads:

[0085] (1) Calculation of road width The road width is greater than 1.5 times the width of the driverless car: The road width calculation method is as follows: Grid the road point cloud and generate a grid map. At the same time, perform 3D polynomial curve fitting on the road point cloud, and draw a straight line perpendicular to the tangent line of a point on the curve. The straight line and the grid map are compared to two points. The Euclidean distance between the two points is the road width.

[0086] (2) Flat road: The road point cloud is segmented and the PCL point cloud library is used to fit the road point cloud to a plane. The distance from the point to the plane is less than the threshold. Points greater than the threshold are clustered and the minimum bounding box size is calculated using the OBB algorithm in PCL and marked as obstacles or potholes.

[0087] (3) Road continuity: The values ​​of the road point cloud in the ground direction are continuous. Discontinuous areas are marked to indicate road interruption.

[0088] S600, path planning: determining the shortest path based on the screened drivable roads as the target planning path.

[0089] In this step, the shortest path is calculated. Specifically, the present invention adopts A* global path planning and Bessel local path planning as the path planning scheme. The A* algorithm can reduce the number of searched nodes while ensuring the shortest path is found. The Bessel curve can ensure the curvature continuity of the local path planning curve. On the basis of ensuring the shortest path, the desired trajectory that is as smooth as possible is planned for the vehicle.

[0090] In summary, it can be seen that the present invention associates the 3D point cloud generated by SLAM technology and the semantic 2D image generated by deep learning technology, extracts the 3D information of the road, and determines whether the road is passable, providing a guarantee for unmanned vehicle path planning and rapid arrival at the disaster site.

[0091] The present application also provides a drone-coordinated unmanned vehicle path planning system, including:

[0092] The calibration module is used to synchronize the time of the sensors of the drone and the unmanned vehicle, and to calibrate the external parameters between the lidar, camera, and inertial navigation system;

[0093] Build a point cloud module, which is used to perform self-positioning and build a three-dimensional point cloud map through the SLAM algorithm, and store the point cloud data in PCD format;

[0094] The semantic segmentation module is used to classify and identify image pixels of drone-collected images using deep learning algorithms;

[0095] The association module is used to convert the point cloud map from the global coordinate system to the lidar coordinate system, and use the external parameters obtained by calibration to convert the point cloud data to the camera coordinate system; through the camera projection model, each point in the point cloud is projected to the corresponding pixel of the image;

[0096] A screening module, used to screen the drivable roads in the projected point cloud map;

[0097] The output module is used to determine the shortest path as the target planning path based on the screened drivable roads.

[0098] For the specific limitations of the unmanned vehicle path planning system, please refer to the limitations of the unmanned vehicle path planning for drone collaboration mentioned above, which will not be repeated here. Each module in the above-mentioned unmanned vehicle path planning system can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0099] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A method for unmanned vehicle path planning in collaboration with unmanned aerial vehicles, characterized in that: The method comprises: S100, UAV sensor spatiotemporal calibration: synchronize the time of the UAV and the unmanned vehicle sensors, and calibrate the external parameters between the lidar, camera, and inertial navigation; S200, SLAM builds point cloud map: uses SLAM algorithm to perform self-positioning and build 3D point cloud map, and stores point cloud data in PCD format; S300, Image Semantic Segmentation: Apply deep learning algorithms to classify and identify image pixels in images collected by drones; S400, image semantics and point cloud map association: convert the point cloud map from the global coordinate system to the lidar coordinate system, and use the external parameters obtained by calibration to convert the point cloud data to the camera coordinate system; through the camera projection model, each point in the point cloud is projected onto the corresponding pixel of the image; S500, screening drivable roads: screening drivable roads in the projected point cloud map; S600, path planning: determining the shortest path based on the screened drivable roads as the target planning path.

2. The unmanned vehicle path planning method according to claim 1, characterized in that: S100 synchronizes the time between the drone and the unmanned vehicle sensors, including: Use RTK's PPS+GPRMC to synchronize time with the drone domain controller. At the same time, use PTP to eliminate the time delay caused by network transmission and accurately obtain the timestamps of all sensors.

3. The unmanned vehicle path planning method according to claim 1, characterized in that: The external parameters between the laser radar, camera, and inertial navigation in S100 are calibrated, including: Pre-calibrate the external parameters between the camera and the lidar, and between the lidar and the inertial navigation system; When the drone is stationary, the checkerboard calibration plate is moved multiple times. By simultaneously capturing the images of the calibration plate in the lidar and camera, the spatial transformation relationship between the two is calculated, that is, the external parameters between the lidar and the camera. When the UAV is moving, the recursive trajectories of the inertial navigation and lidar are calculated respectively, and the external parameters between the lidar and inertial navigation are calculated through the hand-eye calibration principle.

4. The unmanned vehicle path planning method according to claim 1, characterized in that: The S200 uses the SLAM algorithm to locate itself and build a 3D point cloud map, including: The GTSAM optimizer is used to optimize the pose graph, and the optimization objective function is: Where X is the desired lidar keyframe pose, including rotation and translation, t i is the translation component in X, T ij , are the estimated and observed values ​​of the relative pose between the i-th frame and the j-th frame, respectively. I, L, Loop, and G are the key frames corresponding to the IMU pre-integration constraint, the lidar relative pose constraint, the closed-loop constraint, and the GPS constraint, respectively.

5. The unmanned vehicle path planning method according to claim 1, characterized in that: The S300 uses deep learning algorithms to classify and identify image pixels from drone-collected images, including: A fully convolutional network is used to classify each pixel in the image; wherein the classification categories include at least soil, buildings, vegetation, water, roads, and sidewalks; And the edge detection method is used to identify the boundary lines in the image.

6. The unmanned vehicle path planning method according to claim 5, characterized in that: The edge detection method is used to identify the local changes in pixel intensity in the image using Sobel, Canny, and Prwitt to identify the boundary line, including: Preprocess the image to enhance edge features and reduce noise interference; Apply a specific convolution kernel or multi-stage detection process to calculate the gradient size and direction of each pixel in the image; the gradient information reflects the rapid changes in image brightness; Refine the edges through the non-maximum suppression step to ensure that the edges are as thin as possible; Extract clear edge contours from the gradient image to complete the edge recognition process.

7. The unmanned vehicle path planning method according to claim 1, characterized in that: In S400, the point cloud data is converted into the camera coordinate system using the external parameters obtained by calibration; Through the camera projection model, each point in the point cloud is projected onto the corresponding pixel of the image, including: The point cloud data is further converted from the laser radar coordinate system to the camera coordinate system using the external parameters obtained during the calibration process; wherein the external parameters include the rotation matrix and the translation vector; A pinhole projection model of the camera is applied to project each point in the point cloud onto a corresponding pixel of the image; wherein the camera internal parameters of the pinhole projection model include focal length and optical center position.

8. The unmanned vehicle path planning method according to claim 1, characterized in that: The conditions for screening the drivable roads in the projected point cloud map in S500 include at least: Calculate the road width. The road width is greater than 1.5 times the width of the driverless car. The road width is calculated by gridding the road point cloud to generate a grid map. At the same time, a 3D polynomial curve is fitted to the road point cloud. A straight line is drawn perpendicular to the tangent of a point on the curve. The Euclidean distance between the straight line and the grid map compared to the two points is determined as the road width. Flat road: The road point cloud is processed in segments, and the PCL point cloud library is used to fit the road point cloud to a plane. The distance from the point to the plane is less than the threshold, and the points greater than the threshold are clustered. The OBB algorithm in PCL is used to calculate the minimum bounding box size and mark it as an obstacle or pothole. Road continuity: The values ​​of the road point cloud in the ground direction are continuous, and discontinuous areas are marked to indicate road interruption.

9. The unmanned vehicle path planning method according to claim 1, characterized in that: In S600, the shortest path is determined based on the screened drivable roads as the target planning path, including: The path planning schemes adopted are A* global path planning and Bessel local path planning; among them, the A* algorithm can reduce the number of search nodes while ensuring the shortest path; the Bessel curve can ensure the curvature continuity of the local path planning curve, and plan the smoothest possible desired trajectory for the vehicle while ensuring the shortest path.

10. A UAV-coordinated unmanned vehicle path planning system, characterized in that: The system comprises: The calibration module is used to synchronize the time of the sensors of the drone and the unmanned vehicle, and to calibrate the external parameters between the lidar, camera, and inertial navigation system; Build a point cloud module, which is used to perform self-positioning and build a three-dimensional point cloud map through the SLAM algorithm, and store the point cloud data in PCD format; The semantic segmentation module is used to classify and identify image pixels of drone-collected images using deep learning algorithms; The association module is used to convert the point cloud map from the global coordinate system to the lidar coordinate system, and use the external parameters obtained by calibration to convert the point cloud data to the camera coordinate system; through the camera projection model, each point in the point cloud is projected to the corresponding pixel of the image; The screening module is used to screen the drivable roads of the projected point cloud map; the output module is used to determine the shortest path as the target planning path based on the screened drivable roads.

Images