Flying car obstacle avoidance method, system and device based on multi-sensor fusion

Abstract

The invention discloses a flying car obstacle avoidance method, system and device based on multi-sensor fusion, and belongs to the technical field of flying cars, and the obstacle avoidance method comprises the following steps: obtaining image data and point cloud data of the surrounding environment of a flying car after a camera sensor and a laser radar sensor are subjected to time-space synchronization; after the image data and the point cloud data are preprocessed, target detection is carried out respectively to obtain two-dimensional image target information and a point cloud target detection result; after data association is carried out on the two-dimensional image target information and the point cloud target detection result by adopting an IOU-based matching strategy, optimal matching is carried out through a KM association matching algorithm to obtain a matching result and confidence; and performing optimal path planning according to the matching result and the confidence coefficient to obtain an optimal track, and performing obstacle avoidance by the hovercar according to the optimal track. According to the invention, the obstacle detection accuracy of the existing obstacle avoidance technology can be further improved, and the optimal obstacle avoidance track can be planned.

Description

Technical Field

[0001] The present invention belongs to the technical field of flying cars, and in particular relates to a flying car obstacle avoidance method, system and obstacle avoidance device based on multi-sensor fusion. Background Art

[0002] Flying cars not only integrate the comfort and convenience of traditional cars, but also draw on the flight capabilities of aviation technology. However, to achieve safe and autonomous flight of flying cars, the most important technical challenge is how to effectively avoid obstacles and ensure safe navigation in complex and changing airspace environments. Therefore, the obstacle avoidance system, as one of the core modules in the development of flying cars, has a very wide range of applications in flying cars.

[0003] The obstacle avoidance system detects, analyzes and determines obstacles using various technologies to achieve comprehensive and accurate detection of the surrounding environment. Among them, environmental perception technology is one of the key technologies in the obstacle avoidance system. Compared with single sensor target detection, multi-sensor fusion technology has better accuracy and reliability. Multi-sensor fusion technology uses multiple sensors to collect raw data, and combines data processing, information fusion and other technologies to detect the appearance, distance, angle, etc. of obstacles. However, due to the many influencing factors of multi-sensor fusion, there are currently shortcomings such as complex system structure, high cost, and inability to avoid obstacles in the optimal route after they are discovered. Summary of the invention

[0004] The purpose of the present invention is to provide a flying car obstacle avoidance method, system and obstacle avoidance device based on multi-sensor fusion, which can further improve the accuracy of obstacle detection of existing obstacle avoidance technology and plan the optimal obstacle avoidance trajectory.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, a flying car obstacle avoidance method based on multi-sensor fusion comprises the following steps: After the camera sensor and the lidar sensor are synchronized in time and space, the image data and point cloud data of the flying car's surrounding environment are obtained, and after preprocessing the image data and the point cloud data, target detection is performed to obtain two-dimensional image target information and point cloud target detection results respectively; The IOU-based matching strategy is used to first associate the two-dimensional image target information with the point cloud target detection result, and then the KM association matching algorithm is used to perform optimal matching to obtain the matching result and confidence level; An optimal path planning is performed according to the matching result and the confidence level to obtain an optimal trajectory, and the flying car avoids obstacles according to the optimal trajectory.

[0006] In some implementations, after preprocessing the image data and the point cloud data, the steps of respectively performing target detection to obtain two-dimensional image target information and point cloud target detection results specifically include: After labeling and data enhancement of the image data, target detection is performed using the YOLOv8 model to obtain two-dimensional image target information and a two-dimensional detection frame; After dividing the effective area of ​​the point cloud data, voxel filtering is performed on the point cloud data in the effective area, and then the pointpillars algorithm is used to perform target detection to obtain a point cloud target detection result and a three-dimensional detection frame.

[0007] In some implementations, after data association is performed on the two-dimensional image target information and the point cloud target detection result using an IOU-based matching strategy, the steps of performing optimal matching using a KM association matching algorithm to obtain a matching result and a confidence level specifically include: An IOU-based matching strategy is used to calculate the IOU value between the target frame of each of the two-dimensional image target information and the target frame of the corresponding point cloud target detection result. When the IOU value is less than a preset IOU threshold, the two-dimensional image target information or the point cloud target detection result is selected and output according to the surrounding environment of the flying car. When the IOU value is greater than or equal to the preset IOU threshold, the KM association matching algorithm is used to optimally match the two-dimensional image target information and the point cloud target detection result and calculate the confidence to obtain the matching result and its corresponding confidence. The weights of the KM association matching algorithm are the overlap difference and distance difference between the two-dimensional detection frame and the three-dimensional detection frame.

[0008] In some embodiments, when the IOU value is equal to zero, the two-dimensional image target information or the point cloud target detection result is selected and outputted according to the surrounding environment of the flying car.

[0009] In some embodiments, the confidence is calculated using the following formula: (1) in, is the target category weight of the camera, is the target category weight of the laser radar, is the confidence of the target detected by the camera, is the confidence of the target detected by the lidar, is the target confidence after fusion correction.

[0010] In some implementations, the matching results and confidence levels are used to perform optimal path planning using a DWA algorithm to obtain an optimal trajectory.

[0011] In some embodiments, the step of the flying car avoiding obstacles according to the optimal trajectory specifically includes: According to the matching result, it is determined whether there is an obstacle at the preset safety distance of the flying car. If there is an obstacle, the obstacle is avoided according to the optimal trajectory; if there is no obstacle, the flying car continues to travel along the original trajectory.

[0012] In some embodiments, the preset safety distance includes an expected safety distance, a limit safety distance, and an intermediate safety distance; If the distance between the obstacle and the flying car is greater than or equal to the expected safety distance, the flying car continues to travel along the optimal trajectory; If the distance between the obstacle and the flying car is less than the limit safety distance, the flying car will avoid the obstacle; If the distance between the obstacle and the flying car is within the range of the intermediate safety distance, the flying car will issue an early warning.

[0013] In the second aspect, a flying car obstacle avoidance system based on multi-sensor fusion includes: The data acquisition and processing module is used to synchronize the camera sensor and the lidar sensor in time and space, obtain the image data and point cloud data of the flying car's surrounding environment, and perform target detection on the image data and point cloud data after preprocessing, and obtain two-dimensional image target information and point cloud target detection results respectively; A data fusion module is used to associate the two-dimensional image target information and the point cloud target detection result with an IOU-based matching strategy, and then perform optimal matching through a KM association matching algorithm to obtain a matching result and a confidence level; The obstacle avoidance module is used to perform optimal path planning according to the matching result and the confidence level to obtain an optimal trajectory, and the flying car avoids obstacles according to the optimal trajectory.

[0014] In a third aspect, an obstacle avoidance device based on multi-sensor fusion, based on the obstacle avoidance method, comprises: a flying car body, a camera sensor, a lidar sensor, a fixing bracket, a user display, a flying car thrust mechanism, a data processor, an altitude control mechanism and a steering control mechanism; The camera sensor and the laser radar sensor are installed on the main body of the flying car. The camera sensor is centered on the laser radar sensor and is distributed around the laser radar sensor at equal central angles. The camera sensor and the laser radar sensor are fixed together by the fixing bracket. The camera sensor is used to obtain image data of the flying car's surrounding environment and transmit it to the data processor. The laser radar sensor is used to obtain point cloud data of the flying car's surrounding environment and transmit it to the data processor. The user display is installed in the cockpit of the flying car body and is used to display the working status of the flying car; The data processor is installed inside the flying car body, and the flying car obstacle avoidance method is implemented when the data processor executes the computer program. The data processor is installed inside the flying car body, and is used to pre-process the image data and point cloud data and detect targets, plan the best trajectory through data fusion, and control the flying car to avoid obstacles according to the best trajectory; The flying car thrust mechanism is installed at the front and rear ends of the flying car body, and is used to provide the flying car with acceleration thrust and deceleration thrust when avoiding obstacles; The height control mechanism is fixedly installed around the main body of the flying car and is used to control the driving height of the flying car when avoiding obstacles; The steering control mechanism is installed at the fixed connection between the altitude control mechanism and the flying car body, and is used to control the direction of the flying car when avoiding obstacles.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a flying car obstacle avoidance method based on multi-sensor fusion, which first associates the two-dimensional image target information and the point cloud target detection result with data by adopting an IOU-based matching strategy, and then performs optimal matching by using a KM association matching algorithm to obtain a matching result and a confidence level; performs optimal path planning according to the matching result and the confidence level to obtain an optimal trajectory, and the flying car avoids obstacles according to the optimal trajectory. The method can more comprehensively perceive the environment around the flying car, more effectively match the two-dimensional image target information and the three-dimensional point cloud target detection result, reduce false positives and false negatives, and improve the accuracy of the flying car's obstacle avoidance; the method takes into account the obstacle avoidance requirements under different environmental conditions, combines multiple sensor data, can maintain stable obstacle avoidance performance in complex environments, improves the accuracy of obstacle detection, effectively reduces the impact of weather and light on data, and improves the safety and reliability of the flying car.

[0016] Furthermore, the present invention only outputs two-dimensional image target information or point cloud target detection results when the IOU value is less than a preset IOU threshold. When the IOU value is greater than or equal to the preset IOU threshold, the KM association matching algorithm is used to optimally match the two-dimensional image target information and the point cloud target detection results. The two-dimensional image target information or the point cloud target detection results are output according to the surrounding environment of the flying car, which can further enhance the environmental adaptability of the obstacle avoidance method provided by the present invention.

[0017] Furthermore, the present invention adopts the DWA algorithm for optimal path planning based on the fused target detection results and confidence levels, which can obtain a more reasonable and safe flight trajectory and improve the obstacle avoidance efficiency and safety of the flying car.

[0018] Furthermore, during the obstacle avoidance process, the present invention calculates the distance between the obstacle and the flying car, and determines whether the flying car should avoid the obstacle based on a preset safety distance, thereby ensuring that the flying car can avoid the obstacle in time when encountering an obstacle. In addition, an intermediate safety distance is set for early warning, which can remind the driver to make preparations for obstacle avoidance and improve the safety of the flight process. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the structure of an obstacle avoidance device based on multi-sensor fusion provided in Example 2; Figure 2 A schematic diagram of the structure of the steering, acceleration and deceleration control mechanism used in the obstacle avoidance device based on multi-sensor fusion provided in Example 2; Figure 3 The overall flow chart of the flying car obstacle avoidance system based on multi-sensor fusion provided in Example 1; Figure 4 A multi-sensor data fusion flow chart provided in Example 1; Figure 5 Obstacle avoidance flow chart provided for Example 1; Figure 6 An algorithm flow chart for fusing image data and point cloud data provided in Example 1; Figure 7 A flowchart for planning the optimal trajectory provided in Example 1; Figure 8 This is a graph of the algorithm detection results of the YOLO model used in Example 1; Fig. 9 The detection frame detected by the laser radar sensor used in the first embodiment; Fig.10 The detection result after the fusion of the two-dimensional image target information and the point cloud target detection result of the first embodiment; Fig.11 A schematic diagram of an obstacle avoidance method for a flying car based on multi-sensor fusion provided in Example 1; In the figure, 1. Flying car body; 2. Camera sensor; 3. LiDAR sensor; 4. Fixed bracket; 5. User display; 6. Flying car thrust mechanism; 7. Data processor; 8. Altitude control mechanism; 9. Steering control mechanism. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings, and the described content is intended to explain the present invention rather than to limit it.

[0021] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or are inherent to these processes, methods, systems, products or apparatus.

[0022] Embodiment 1 like Figure 3 , Figure 4 As shown, this embodiment provides a flying car obstacle avoidance method based on multi-sensor fusion, comprising the following steps: Step 0: Multi-sensor installation: First, focus on the installation and configuration of multi-sensors and related modules to provide the hardware foundation for the subsequent obstacle avoidance system to operate efficiently and accurately.

[0023] Step 1: Install the camera sensor and lidar sensor around the flying car (such as Figure 1 ), make sure these sensors are mounted securely and at the correct angle to get the most comprehensive and clear data.

[0024] Step 2: Install the data processing module, obstacle avoidance module and user display inside the flying car.

[0025] Step 3: Ensure that all hardware devices are connected stably, interfaces are matched, and power supply is sufficient and reliable. This provides the necessary hardware foundation for subsequent target detection.

[0026] Step 1: Collect and process image data. First, collect the original image data of the flying car's surrounding environment to provide necessary data information for subsequent system data processing and decision-making systems.

[0027] The first step is to collect a large amount of image data of different types of environmental data under the camera sensor. For each environment, take sample photos under different angles and lighting conditions as much as possible to improve the diversity of the original data. For example, take 10,000 original image data around the predetermined road.

[0028] In the second step, the camera sensor is used to collect image data and obtain JPG format images. Then, the Labelimg software is used to annotate the training set of image data and generate an extensible markup language (XML) file containing the image file name, path, marked target type, and bounding box location information. Since the YOLOv8 algorithm uses the label file format, the XML label file needs to be converted to the label file format.

[0029] In step 3, considering the common transformations of objects in the test environment in this embodiment, three methods of random rotation, color transformation and random noise are used for data enhancement. For example, for each sample photo and each enhancement method, one variant will be generated, so the total number of samples after data enhancement is 40,000. This method increases the number and diversity of samples, makes the training effect better, and improves the robustness of the model.

[0030] Step 4, the YOLO model is a fast and accurate real-time object detection system that provides high accuracy and efficiency while maintaining detection speed. The basic model used in this embodiment for target detection on image data is the YOLOv8 model. In model training, the image data collected and enhanced by the camera is distributed in a ratio of 8:2, of which 80% is used as a training set and 20% is used as a test set. Start training the training set, and train the training set for 300 cycles to obtain the optimal target detection model.

[0031] Step 5: Use the prepared test set to evaluate and verify the trained detection model YOLOv8 to ensure that the model performance meets the requirements. Use the trained YOLOv8 model to perform target detection and obtain the two-dimensional image target information.

[0032] Step 2: Use lidar sensors to collect and process point cloud data of the flying car's surrounding environment.

[0033] The first step is to collect a large amount of point cloud data from lidar sensors in different environments, and take sample point clouds under different environmental conditions for each environment as much as possible to improve the diversity of the original data. For example, 10,000 frames of discontinuous original point cloud data are taken around the predetermined road.

[0034] Step 2: divide the point cloud data collected by the LiDAR sensor into effective areas. For example, if the point cloud density 100 meters in front of the radar is too low to accurately present effective information about obstacles, the point cloud data 100 meters away should be removed, and the point cloud data within 20 meters behind the radar should be retained. Therefore, the range of the flying car in the X direction is (-20,100). The point cloud data within 10 meters on both sides of the Y axis of the flying car is retained, that is, the range in the Y direction is (-10,10), and the range in the Z axis direction is (-2,5).

[0035] Step 3: Filter the point cloud data. Voxel filtering is performed on the point cloud data to remove some dense and useless points in the original point cloud, while reducing the number of point clouds without affecting the geometric structure characteristics of the objects in the point cloud.

[0036] Step 4: This embodiment uses a voxel feature-based target detection method for point cloud feature extraction and detection. In model training, the data collected by the lidar sensor is distributed in a 1:1 ratio, with 50% as a training set and 50% as a validation set. Using the Pointpillars algorithm, the collected point cloud data is used for model training and validation, and finally the trained model is used to perform target detection on the point cloud data to obtain the point cloud target detection result.

[0037] Step 3: Multi-sensor decision-level fusion, which combines the two-dimensional image target information detected by vision and the point cloud target detection results obtained based on the lidar algorithm at the decision level.

[0038] In the first step, the 3D point cloud target detection results of the LiDAR are displayed on the 2D image. After the LiDAR and the camera are synchronized and calibrated in time, the point cloud target detection results correspond to the targets on the 2D image target information. The LiDAR and the camera are calibrated in pairs to obtain the positional relationship between the camera and the LiDAR. The camera intrinsic parameters are calibrated using the Matlab visual calibration toolbox, and the spatial coordinates of the LiDAR are converted to the spatial coordinates of the camera through the coordinate matrix. Autoware is used for joint calibration.

[0039] Step 2, such as Figure 8~Figure 10 As shown, the two-dimensional image target information and the point cloud target detection results are data associated. The intersection-over-union (IOU) matching strategy is used to effectively associate the two sets of independent detection results, and the IOU value between the target box of each two-dimensional image target information and the target box of its corresponding point cloud target detection result is calculated. Then, the KM association matching algorithm is used to optimally match the camera and radar detection targets (i.e., the two-dimensional image target information and the point cloud target detection results); The IOU matching strategy is used to associate the two-dimensional image target information and point cloud target detection results. There are the following problems: (1) The lidar detection frame may be smaller than the camera detection frame, resulting in a smaller calculated IOU value, which affects the effect of data fusion; (2) How to set an appropriate IOU threshold to distinguish positive and negative samples? Too high a threshold will cause many reasonable matching results to be ignored, and too low a threshold will cause a large number of false matches. Therefore, in order to solve the problems of the IOU matching strategy, the following method is used: (1) The ratio of the intersection area of ​​the three-dimensional detection frame detected by the lidar and the two-dimensional detection frame recognized by the camera to the area of ​​the two-dimensional detection frame recognized by the camera is used as an indicator of the matching probability; (2) The target frame intersection-union ratio indicator and spatial distance are used for data association. The preset IOU threshold is set to 0.3 to judge the matching results.

[0040] When using the KM association matching algorithm, the weights of the vertices need to be constantly adjusted to ensure that the optimal match is found. However, this process requires precise control, otherwise it is easy to fall into a local optimal solution. Therefore, the YOLOv8 target detection algorithm is used to obtain a two-dimensional box target set on the image. , the three-dimensional detection box is obtained through the above point cloud target detection and projected onto the image to generate a two-dimensional detection box set , the matching of these two detection frames is regarded as a bipartite graph problem to find the optimal matching result, and the difference between the two-dimensional detection frame and the three-dimensional detection frame is used as the weight of the KM association matching algorithm.

[0041] Step 3: Output the matching results and confidence level. Figure 6 As shown in the figure, when IOU<0.3, it is determined that only the camera or the laser radar detects the target. Then, the two-dimensional image target information or the point cloud target detection result is selected according to the surrounding environment of the flying car. Specifically, when encountering strong light or direct sunlight, the image may be overexposed, and at night or on cloudy days, the image may be too dark. At this time, using the camera for obstacle detection may cause detection failure or missed detection. In this case, only the laser radar can detect the target, and the detection results of the camera and the laser radar cannot be effectively fused. Therefore, the point cloud target detection result of the laser radar point cloud data clustering is directly used as the final obstacle detection result; when the target is at a long distance, the acquired point cloud data will become sparse, which makes it difficult to effectively cluster these sparse point clouds, and then the laser radar may not be able to detect some targets. The camera can still recognize the target in this case. In this case, due to the lack or insufficiency of the laser radar data, it cannot be effectively fused with the camera data, so the camera's two-dimensional image target information is directly used as the final detection output. When IOU=0, the same treatment is followed; When IOU ≥ 0.3, the vehicle in front is within 50m, and the two bounding boxes are considered to match when the matching distance threshold is less than 0.4m; when IOU is greater than or equal to 0.3, within 100m, and the two bounding boxes are considered to match when the matching distance threshold is less than 1m, the two bounding boxes are considered to match, so it is determined that the camera and lidar detect the target at the same time, and the KM association matching algorithm is used to optimally match the two-dimensional image target information and the point cloud target detection results and calculate the confidence to obtain the matching result and its corresponding confidence. The weighted average method is used to fuse and correct their confidences. The weighted average formula is: (1) In formula (1), is the target category weight of the camera, is the target category weight of the laser radar, is the confidence of the target detected by the camera, is the confidence of the target detected by the lidar, is the target confidence after fusion correction.

[0042] Step 4: Local path planning: Figure 7 As shown, when there are obstacles, the system uses the DWA algorithm to plan the optimal route to avoid more serious dangers.

[0043] The first step is to perform speed space sampling. Due to the hardware, structural and environmental constraints of the flying car, three factors need to be considered: speed limit, acceleration limit and environmental obstacle limit. The following three formulas are used to limit the speed. Finally, the flying car speed sampling space is the intersection of the three speed spaces.

[0044] (2) (3) (4) (5) The second step is to predict the trajectory by using the kinematic model using the following formula.

[0045] (6) (7) (8) Step 3: Evaluate the trajectories and select the best one. Use the following formula to evaluate the trajectories.

[0046] (9) Step 4: Perform normalization.

[0047] (10) (11) (12) The DWA algorithm also needs to combine the matching results and confidence obtained in step 3 to calculate the optimal trajectory.

[0048] Step 5: Early warning module: Avoid obstacles according to the planned optimal trajectory, and issue an early warning to remind the driver to drive along the optimal trajectory.

[0049] Step 1: The optimal trajectory is input into the user display interface to remind the driver to drive according to the trajectory.

[0050] Step 2: Turn on the warning module to remind the driver to respond as soon as possible and control the flying car according to the optimal trajectory. After the optimal trajectory is input into the display interface, the display interface flashes the LED light to remind the driver to control.

[0051] Step 3, such as Figure 5 As shown, it is determined whether there is an obstacle at the preset safety distance of the flying car according to the matching result. If there is an obstacle, the obstacle is avoided according to the optimal trajectory; if there is no obstacle, the flying car continues to travel along the original trajectory.

[0052] The preset safety distance includes an expected safety distance, a limit safety distance and an intermediate safety distance; if the distance between the obstacle and the flying car is greater than or equal to the expected safety distance, the flying car continues to travel according to the optimal trajectory; if the distance between the obstacle and the flying car is less than the limit safety distance, the flying car avoids the obstacle; if the distance between the obstacle and the flying car is within the range of the intermediate safety distance, the flying car issues an early warning.

[0053] If the distance between the obstacle and the flying car is less than the maximum safety distance and the driver still does not respond, emergency obstacle avoidance will be initiated. When the distance between the obstacle and the flying car is less than the maximum safety distance of 30 meters, emergency obstacle avoidance will be performed.

[0054] Step 6: Control unit: Fig.11 As shown, after the DWA algorithm selects the optimal path, it controls the flying car according to the optimal trajectory.

[0055] Step 1: Control the flying speed of the flying car according to the obstacle avoidance flight path. If the optimal trajectory in step 4 requires acceleration or deceleration to pass the current obstacle, the flying car thrust mechanism is used to control the speed to achieve the result.

[0056] Step 2: Control the flight direction of the flying car according to the obstacle avoidance flight path. If the optimal trajectory in step 4 requires turning left or right to pass the current obstacle, the steering control mechanism is used to perform steering control to achieve the result.

[0057] Step 3: Control the flying altitude of the flying car according to the obstacle avoidance flight path. If the optimal trajectory in step 4 needs to be raised and lowered to pass the current obstacle, the altitude control mechanism is used to control the altitude to achieve the result.

[0058] The present embodiment also provides a flying car obstacle avoidance system based on multi-sensor fusion, comprising: a data acquisition and processing module, which is used to synchronize the camera sensor and the laser radar sensor in time and space, obtain image data and point cloud data of the flying car's surrounding environment, and perform target detection on the image data and point cloud data after preprocessing, and obtain two-dimensional image target information and point cloud target detection results respectively; A data fusion module is used to associate the two-dimensional image target information and the point cloud target detection result with an IOU-based matching strategy, and then perform optimal matching through a KM association matching algorithm to obtain a matching result and a confidence level; The obstacle avoidance module is used to perform optimal path planning according to the matching result and the confidence level to obtain an optimal trajectory, and the flying car avoids obstacles according to the optimal trajectory.

[0059] Embodiment 2 like Figure 1 and Figure 2 As shown, the obstacle avoidance device based on multi-sensor fusion provided in this embodiment includes: a flying car body 1, a camera sensor 2, a lidar sensor 3, a fixed bracket 4, a user display 5, a flying car thrust mechanism 6, a data processor 7, an altitude control mechanism 8, and a steering control mechanism 9.

[0060] The obstacle avoidance system of the flying car of the present invention comprises a flying car body 1, which includes a computer etc. required for the operation of the system. The camera sensor 2 is installed around the laser radar sensor 3 according to the distribution of equal central angles with the laser radar sensor 3 as the center, and the camera sensor 2 and the laser radar sensor 3 are fixed together by a fixing bracket 4. The user display 5 is installed in the cockpit of the flying car, and is mainly used to display the working status of the whole system. The flying car thrust mechanism 6 is installed at the front and rear of the flying car body 1, and mainly provides the thrust for the flying car to move forward when avoiding obstacles and the thrust for acceleration and deceleration when avoiding obstacles. The camera sensor 2 is used to obtain the image data of the flying car's surrounding environment and transmit it to the data processor 7, and the laser radar sensor 3 is used to obtain the point cloud data of the flying car's surrounding environment and transmit it to the data processor 7. The data processor 7 is installed at the rear of the flying car, and is mainly used to fuse data information and determine whether to perform obstacle avoidance processing. The height control mechanism 8 is installed around the flying car body 1, and is used to control the height of the flying car when avoiding obstacles. The steering control mechanism 9 is installed on the fixing frame of the height control mechanism 8, and is used to control the steering of the flying car when avoiding obstacles. When the camera sensor 2 and the laser radar sensor 3 detect the target result, they are input into the data processor 7 for data fusion and selection of the optimal route planning for obstacle avoidance processing. The optimal path is output to the user display 5 to remind the driver to start the flying car thrust mechanism 6, the altitude control mechanism 8 and the steering control mechanism 9 to take corresponding obstacle avoidance measures. It should be noted that the flying car uses propellers for movement, altitude control and steering when it is driving normally. When encountering a situation where emergency obstacle avoidance is required, the flying car thrust mechanism, altitude control mechanism and steering control mechanism in the obstacle avoidance device provided in this embodiment are used for movement, altitude control and steering.

[0061] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the specific implementation modes of the present invention can still be modified or replaced by equivalents, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A flying car obstacle avoidance method based on multi-sensor fusion, characterized in that: The following steps are involved: After the camera sensor and the lidar sensor are synchronized in time and space, the image data and point cloud data of the flying car's surrounding environment are obtained, and after preprocessing the image data and the point cloud data, target detection is performed to obtain two-dimensional image target information and point cloud target detection results respectively; After data association is performed on the two-dimensional image target information and the point cloud target detection result by using an IOU-based matching strategy, the KM association matching algorithm is used to perform optimal matching to obtain the matching result and confidence level; An optimal path planning is performed according to the matching result and the confidence level to obtain an optimal trajectory, and the flying car avoids obstacles according to the optimal trajectory.

2. The flying car obstacle avoidance method according to claim 1, characterized in that: After preprocessing the image data and the point cloud data, the steps of respectively performing target detection to obtain two-dimensional image target information and point cloud target detection results specifically include: After labeling and data enhancement of the image data, target detection is performed using the YOLOv8 model to obtain two-dimensional image target information and a two-dimensional detection frame; After dividing the effective area of ​​the point cloud data, voxel filtering is performed on the point cloud data in the effective area, and then the pointpillars algorithm is used to perform target detection to obtain a point cloud target detection result and a three-dimensional detection frame.

3. The flying car obstacle avoidance method according to claim 2, characterized in that: After data association is performed on the two-dimensional image target information and the point cloud target detection result by using an IOU-based matching strategy, the steps of performing optimal matching by using a KM association matching algorithm to obtain a matching result and a confidence level specifically include: An IOU-based matching strategy is used to calculate the IOU value between the target frame of each of the two-dimensional image target information and the target frame of the corresponding point cloud target detection result. When the IOU value is less than a preset IOU threshold, the two-dimensional image target information or the point cloud target detection result is selected and output according to the surrounding environment of the flying car. When the IOU value is greater than or equal to the preset IOU threshold, the KM association matching algorithm is used to optimally match the two-dimensional image target information and the point cloud target detection result and calculate the confidence to obtain the matching result and its corresponding confidence. The weights of the KM association matching algorithm are the overlap difference and distance difference between the two-dimensional detection frame and the three-dimensional detection frame.

4. The flying car obstacle avoidance method according to claim 3, characterized in that: When the IOU value is equal to zero, the two-dimensional image target information or point cloud target detection result is output according to the surrounding environment of the flying car.

5. The flying car obstacle avoidance method according to claim 3, characterized in that: The confidence is calculated using the following formula: （1） in, is the target category weight of the camera, is the target category weight of the laser radar, is the confidence of the target detected by the camera, is the confidence of the target detected by the lidar, is the target confidence after fusion correction.

6. The flying car obstacle avoidance method according to claim 1, characterized in that: The matching results and confidence levels are used to perform optimal path planning through the DWA algorithm to obtain the best trajectory.

7. The flying car obstacle avoidance method according to claim 1, characterized in that: The step of the flying car avoiding obstacles according to the optimal trajectory specifically includes: According to the matching result, it is determined whether there is an obstacle at the preset safety distance of the flying car. If there is an obstacle, the obstacle is avoided according to the optimal trajectory; if there is no obstacle, the flying car continues to travel along the original trajectory.

8. The flying car obstacle avoidance method according to claim 7, characterized in that: The preset safety distance includes an expected safety distance, a limit safety distance and an intermediate safety distance; If the distance between the obstacle and the flying car is greater than or equal to the expected safety distance, the flying car continues to travel along the optimal trajectory; If the distance between the obstacle and the flying car is less than the limit safety distance, the flying car will avoid the obstacle; If the distance between the obstacle and the flying car is within the range of the intermediate safety distance, the flying car will issue an early warning.

9. A flying car obstacle avoidance system based on multi-sensor fusion, characterized in that: include: The data acquisition and processing module is used to synchronize the camera sensor and the lidar sensor in time and space, obtain the image data and point cloud data of the flying car's surrounding environment, and perform target detection on the image data and point cloud data after preprocessing, and obtain two-dimensional image target information and point cloud target detection results respectively; A data fusion module is used to associate the two-dimensional image target information and the point cloud target detection result with an IOU-based matching strategy, and then perform optimal matching through a KM association matching algorithm to obtain a matching result and a confidence level; The obstacle avoidance module is used to perform optimal path planning according to the matching result and the confidence level to obtain an optimal trajectory, and the flying car avoids obstacles according to the optimal trajectory.

10. An obstacle avoidance device based on multi-sensor fusion, characterized in that: include: The flying car body, camera sensor, lidar sensor, fixing bracket, user display, flying car thrust mechanism, data processor, altitude control mechanism and steering control mechanism; The camera sensor and the laser radar sensor are installed on the main body of the flying car. The camera sensor is centered on the laser radar sensor and is distributed around the laser radar sensor at equal central angles. The camera sensor and the laser radar sensor are fixed together by the fixing bracket. The camera sensor is used to obtain image data of the flying car's surrounding environment and transmit it to the data processor. The laser radar sensor is used to obtain point cloud data of the flying car's surrounding environment and transmit it to the data processor. The user display is installed in the cockpit of the flying car body and is used to display the working status of the flying car; A computer program is run in the data processor, and when the data processor executes the computer program, the flying car obstacle avoidance method according to any one of claims 1 to 8 is implemented. The data processor is installed inside the flying car body, and is used to plan an optimal trajectory through data fusion after preprocessing and target detection of the image data and point cloud data, and control the flying car to avoid obstacles according to the optimal trajectory; The flying car thrust mechanism is installed at the front and rear ends of the flying car body, and is used to provide the flying car with acceleration thrust and deceleration thrust when avoiding obstacles; The height control mechanism is fixedly installed around the main body of the flying car and is used to control the driving height of the flying car when avoiding obstacles; The steering control mechanism is installed at the fixed connection between the altitude control mechanism and the flying car body, and is used to control the direction of the flying car when avoiding obstacles.

Images