Aerocar mode conversion method, system and equipment and storage medium

Abstract

The invention discloses an aerocar mode conversion method, system and device and a storage medium, and the method comprises the steps: S1, obtaining image data, point cloud data and radar data of the surrounding environment of an aerocar, and carrying out the target detection; s2, fusing target detection results of the image data and the point cloud data; s3, performing confidence correction on the fusion result by using a target detection result of the radar data; and S4, according to the correction result, judging whether the surrounding environment meets the requirement of mode conversion or not, if the judgment result is that the surrounding environment is not shielded, sending out a mode conversion permission signal, and if the judgment result is that the surrounding environment is shielded, sending out a mode conversion forbidding signal. The detection accuracy of the surrounding environment is improved, and stable mode conversion of the hovercar is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of driving monitoring and relates to a flying car mode conversion method, system, device and storage medium. Background Art

[0002] With the rapid development of science and technology, flying cars, as an important part of future urban transportation, have gradually attracted widespread attention. Flying cars have the ability to travel on land and in the air, which can effectively alleviate urban traffic congestion and improve travel efficiency. However, the safe and smooth conversion of flying cars between different flight modes is still a major challenge in technical implementation. In the current related technologies, the mode conversion method mainly relies on manual operation, which is prone to accidents. In addition, there are two main ways for existing flying cars to take off, namely taxiing takeoff and vertical takeoff. However, both methods have their own defects. For example, taxiing takeoff requires a longer runway and has stricter requirements for the site, while the vertical takeoff mode requires the flying car to switch modes at a slower driving speed, and the existing sensor environment detection method cannot meet the accuracy requirements of environmental monitoring when the flying car takes off and lands. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a flying car mode conversion method, system, device and storage medium, which improves the detection accuracy of the surrounding environment and enables the flying car to achieve smooth mode conversion.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions: A flying car mode conversion method includes the following process: S1, obtains image data, point cloud data and radar data of the flying car's surrounding environment, and performs target detection respectively; S2, fuses the target detection results of image data and point cloud data; S3, using the target detection result of radar data to correct the confidence of the fusion result; S4, judging whether the surrounding environment meets the requirements of mode switching according to the correction result, if the judgment result is that there is no obstruction, a signal allowing mode switching is issued, if the judgment result is that there is obstruction, a signal prohibiting mode switching is issued.

[0005] Preferably, in S1, the YOLOV8 model is used to detect targets on image data, and the YOLOV8 model establishment process is: collecting image data of the surrounding environment; annotating the collected original image information; using random rotation, color transformation, random noise and other methods to enhance the data; training the YOLOV8 model used to detect image targets to obtain the optimal model.

[0006] Preferably, in S1, a point cloud data detection model is used to perform target detection on point cloud data based on voxel features, and the process of establishing the point cloud data detection model is: collecting point cloud data of the surrounding environment; dividing the point cloud data into effective areas; performing voxel filtering on the point cloud and downsampling the point cloud; using the Patchwork++ algorithm to segment the point cloud; and using the Pointpillars algorithm to train and verify the model using the collected point cloud data.

[0007] Preferably, in S1, the process of performing target detection on radar data is: collecting radar data of the surrounding environment; performing target clustering on the radar data, using the DBSCAN algorithm to detect outliers, and then distinguishing the outliers and being able to exclude them simultaneously; screening valid targets, preliminarily screening out stationary targets, false targets and empty targets; and performing consistency check and tracking on valid targets.

[0008] Preferably, the specific process of S2 is: displaying the point cloud data on a two-dimensional image, and making one-to-one correspondence between the point cloud data and the targets on the image data; using an evaluation index combining target distance and point cloud image target intersection and union ratio to perform data association to obtain possible matching weights for each target, arranging them, and then using the KM algorithm to obtain the optimal matching result, and outputting the result and confidence level.

[0009] Preferably, the specific process of S3 is: the fused target results and radar data target detection results are screened using angle indicators; the distance is used as a difference judgment indicator for the target matching after screening to obtain the possible matching weight of each target, and the order is arranged according to the weight size; the KM matching algorithm is used to obtain the optimal match between the fused target and the millimeter wave radar target; and the confidence of the matched image, point cloud and radar target is corrected using a weighted average method.

[0010] Preferably, the specific process of S4 is: judging whether the flying car is in flight mode or land mode; if in land mode, judging whether the mode conversion requirement is met according to the correction result, when there are no obstacles within a preset distance around the flying car and the driver starts the control command of the mode conversion, the mode conversion is performed and a taxiing takeoff is performed; if the judgment result is that there is an obstruction in the preset road, judging whether the surrounding environment of the flying car meets the conditions for vertical takeoff, if so, vertical takeoff is performed, and if neither is satisfied, a signal prohibiting mode conversion is issued; if the flying car is in flight mode and the driver performs a landing operation, then according to the correction result, when there are no obstacles within a preset distance around the flying car, the flying car is landed.

[0011] A flying car mode conversion system, comprising: The target detection module is used to obtain image data, point cloud data and radar data of the flying car's surrounding environment and perform target detection respectively; A fusion module is used to fuse the target detection results of image data and point cloud data; A correction module is used to correct the confidence of the fusion result using the target detection result of the radar data; The judgment module is used to judge whether the surrounding environment meets the requirements of mode conversion according to the correction result. If the judgment result is that there is no obstruction, a signal allowing mode conversion is issued; if the judgment result is that there is obstruction, a signal prohibiting mode conversion is issued.

[0012] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the flying car mode conversion method when executing the computer program.

[0013] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the flying car mode conversion method are implemented.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention fuses the image data and point cloud data after target detection, and then identifies and determines the surrounding environment to improve the accuracy of detection. Millimeter-wave radar data has a unique advantage. It is less affected by weather changes and is basically unaffected by sunny days, nighttime, rainy and foggy weather. Therefore, it has the ability to work all day long. Millimeter-wave radar not only has a large detection range, but also has high detection accuracy for the distance and movement speed of the target relative to the radar, and can detect more targets with lower confidence. The multi-sensor data fusion method improves the detection accuracy of the surrounding environment, and can flexibly switch the appropriate take-off method according to different environmental conditions, allowing the flying car to achieve smooth mode conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the multi-sensor arrangement structure of the present invention; Figure 2 This is an overall flow chart of the flying car mode conversion system of the present invention; Figure 3 This is a schematic diagram of the flight mode of the flying car of the present invention; Figure 4 This is a schematic diagram of the taxiing takeoff mode of the flying car of the present invention; Figure 5 This is a schematic diagram of the vertical take-off mode of the flying car of the present invention; Figure 6This is a schematic diagram of the land driving mode of the flying car of the present invention; Figure 7 This is a multi-sensor fusion flow chart of the present invention; Figure 8 This is a mode conversion flow chart of the present invention; Fig. 9 This is the laser radar and camera fusion effect diagram of the present invention; Fig.10 This is a diagram showing the effect of modifying the target confidence of the millimeter-wave radar of the present invention.

[0016] Among them: 1-flying car body, 2-camera, 3-laser, 4-millimeter wave radar, 5-speed sensor, 6-data processor, 7-retractable fixed wings, 8-user interface, 9-foldable ducted fan. DETAILED DESCRIPTION

[0017] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0018] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs. The terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or can communicate with each other; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" used in this article includes any and all combinations of one or more related listed items. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0020] The disclosure below provides many different embodiments or examples to realize different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplicity and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed. In addition, the present invention provides various specific examples of processes and materials, but those of ordinary skill in the art can be aware of the application of other processes and / or the use of other materials.

[0021] like Figure 1As shown, the flying car capable of mode switching according to the present invention comprises a flying car body 1, wherein the flying car body 1 includes a computer etc. required for system operation. The camera 2 is installed around the laser radar 3 in a distribution centered on the laser radar 3, the speed sensor 5 is installed beside the laser radar 3, and the millimeter wave radar 4 is installed at the outermost periphery of the flying car. The user display 9 is installed in the cockpit, mainly used for displaying the working status of the entire system. The retractable fixed wing 7 is installed at the bottom of the flying car body 1, mainly used for taxiing and landing when the space in front is open. The foldable ducted fan 9 is installed in the middle of the top of the flying car, mainly used for vertical take-off and landing when the space in front is not open, and for providing power requirements for land driving and mode switching. The data processor 6 is mainly used for data information fusion and judgment whether to perform mode switching. When the camera 2, the laser radar 3, the millimeter wave radar 4 and the speed sensor 5 detect the target result, they are input into the data processor 6 for data fusion to judge whether mode switching can be performed, and the judgment result is output to the user interface 8. If mode switching can be performed, the control system is started according to the driver's intention to perform mode switching. When the foldable ducted fan 9 fails, the foldable ducted fan 9 on the other side, which is mirror-symmetrical, is stopped accordingly to ensure that the balance is maintained and the direction of the flying car is better controlled.

[0022] Figure 3 In order to adjust the posture of the flying car in the air, the foldable ducted fan 9 is unfolded, and the foldable ducted fan 9 is mainly used to control the navigation of the flying car, and the retractable fixed wing 7 is used to assist the navigation of the flying car.

[0023] Figure 4 For the posture of the flying car when performing a taxiing takeoff, the retractable fixed wing 7 is in an extended state and the foldable ducted fan 9 is in a retracted state, facing rearward, so as to provide sufficient power for the flying car when performing a taxiing takeoff so that the flying car can reach the speed requirement for taxiing takeoff.

[0024] Figure 5 For the posture of the flying car when performing vertical take-off and landing, the retractable fixed wing 7 is in a retracted state, and the foldable ducted fan 9 is in an extended state, facing downward, to provide the flying car with sufficient lifting power when performing a vertical take-off mode to help the flying car achieve the conditions for vertical take-off.

[0025] Figure 6 It is the posture of the flying car when it is driving on land, and it is a posture of the flying car when it is driving on land. The foldable ducted fan 9 and the retractable fixed wing 7 are both in a retracted state.

[0026] Install cameras, lidar, millimeter-wave radar, and speed sensors around the flying car (such as Figure 1As shown in the figure, ensure that these sensors are installed firmly and at the correct angle to obtain the most comprehensive and clear data. Install the data processing module, control module and user display inside the aircraft vehicle. Ensure that all hardware devices are connected stably, the interfaces are matched, and the power supply is sufficient and reliable. Provide the necessary hardware foundation for subsequent target detection.

[0027] like Figure 2 As shown, the flying car mode conversion process is: Step 1, camera raw data collection and processing: First, collect the raw data of the flying car's surrounding environment to provide necessary data information for subsequent system data processing and control system. Step 1, turn on the camera to collect a large amount of image data of the surrounding environment. Step 2, annotate the collected raw image information. Step 3, use random rotation, color transformation, random noise and other methods to enhance data and increase sample diversity. Step 4, train the YOLOV8 model used to detect image targets to obtain the optimal model. Step 5, verify the trained detection model to ensure accuracy and lay the foundation for subsequent data fusion, target recognition and other processing processes. Step 6, obtain image data of the flying car's surrounding environment and perform target detection through the trained YOLOV8 model.

[0028] Step 2: Collection and processing of raw data from LiDAR: Collect raw data from the environment around the flying car to provide necessary data information for subsequent system data processing and control systems. Step 1: Turn on the LiDAR to collect a large amount of point cloud data from the surrounding environment. Step 2: Divide the effective area of ​​the LiDAR point cloud so that the information of obstacles can be accurately presented. Step 3: Perform voxel filtering on the point cloud and downsample the point cloud. Step 4: Segment the point cloud. The Patchwork++ algorithm effectively segments the ground point cloud and non-ground point cloud to adapt to different road scenes. Step 5: Use the Pointpillars algorithm to train and verify the point cloud data detection model using the collected data. Step 6: Obtain point cloud data from the flying car's surrounding environment. The point cloud data detection model performs target detection based on voxel features.

[0029] Step 3, millimeter-wave radar raw data collection and processing: collect raw data of the flying car's surrounding environment to provide necessary data information for subsequent system data processing and control system. Step 1, turn on the millimeter-wave radar to collect a large amount of radar data of the surrounding environment. Step 2, cluster the millimeter-wave radar targets, use the DBSCAN algorithm to detect outliers, and then identify the outliers and exclude them simultaneously. Step 3, screen valid targets, and preliminarily screen out stationary targets, false targets and empty targets. Step 4, perform consistency check and tracking on the millimeter-wave radar valid targets.

[0030] Step 4: Information fusion of camera detection results and lidar detection results: Decision-level fusion of the two-dimensional image target information detected by the camera and the target detection results obtained based on the lidar algorithm. Step 1: Calibrate several sensors. Step 2: Perform data association, and use the evaluation index of target distance and point cloud image target intersection ratio to perform data fusion to reduce the matching error caused by a single indicator. Step 3: Output the results and confidence of the fusion of image detection and point cloud detection data to provide a data basis for subsequent control modules.

[0031] Step 5: Use millimeter-wave radar data to correct the confidence of the detection result: Use millimeter-wave radar data to correct the confidence of the target output by the camera and lidar fusion. Step 1: Use the angle index to filter the fused target result and the filtered millimeter-wave radar target. Step 2: Use the distance as the difference judgment index for the matching of the filtered targets, obtain the possible matching weights of each target, and arrange them in order according to the weight size. Step 3: Use the KM matching algorithm to obtain the optimal match between the fused target and the millimeter-wave radar target. Step 4: Use the weighted average method to correct the confidence of the matched image, point cloud and millimeter-wave radar target.

[0032] Step 6: Perform mode conversion based on the fused data results of the camera, laser radar, and millimeter-wave radar. Step 1: Determine whether mode conversion can be performed based on the fused data results. Step 2: If the judgment result is no obstruction, a signal to allow mode conversion is issued; if the judgment result is obstruction, a signal to prohibit mode conversion is issued; if mode conversion can be performed, determine which method to use for mode conversion based on user needs. Step 3: Output the results to the user display interface.

[0033] like Figure 7 As shown, the specific process of the above method is: Step 1: Image data acquisition and processing: First, collect the original image data of the flying car's surrounding environment to provide necessary data information for subsequent system data processing and decision-making systems.

[0034] The first step is to collect a large amount of image data of different types of environmental data under the camera. For each environment, take sample photos under different angles and lighting conditions as much as possible to improve the diversity of the original data. For example, take 8,000 original image data around the predetermined road.

[0035] Step 2: The image data collected by the camera is obtained in JPG format, and then the training set is annotated using Labelimg software. Then an Extensible Markup Language (XML) file containing the image file name, path, labeled target type, and bounding box location information is generated. Since the YOLOv8 algorithm uses the label file format, the XML label file needs to be converted to the label file format.

[0036] In step 3, considering the common transformations of objects in the test environment in the present invention, three methods of random rotation, color transformation, and random noise are used for data enhancement. For example, one variant will be generated for each sample photo and each enhancement method, so the total number of samples after data enhancement is 32,000. This method increases the number and diversity of samples, makes the training effect better, and improves the robustness of the model.

[0037] Step 4, the YOLO model is a fast and accurate real-time object detection system that provides high accuracy and efficiency while maintaining the detection speed. The basic model used by the present invention for image detection is the YOLOv8 model. In the model training, the image data collected and enhanced by the camera is distributed in a ratio of 8:2, of which 80% is used as a training set and 20% is used as a test set. Start training the training set, and train the training set for 300 cycles to obtain the optimal detection model.

[0038] In step 5, the trained detection model YOLOv8 is evaluated and verified using the prepared test set to ensure that the model performance meets the requirements.

[0039] Step 6: Obtain image data of the flying car’s surroundings and perform target detection using the trained YOLOV8 model.

[0040] Step 2: Collection and processing of lidar point cloud: Collect the original image data of the flying car's surrounding environment to provide necessary data information for subsequent system data processing and decision-making systems.

[0041] The first step is to collect a large amount of laser radar point cloud data in different environments, and take sample point clouds under different environmental conditions for each environment as much as possible to improve the diversity of the original data. For example, take 8,000 frames of discontinuous original point cloud data around the predetermined road.

[0042] Step 2: Divide the effective area of ​​the LiDAR point cloud. For example, if the point cloud density 100 meters in front of the radar is too low to accurately present effective information about obstacles, the point cloud data 100 meters away should be removed, and the point cloud data within 20 meters behind the radar is retained, so the range of the flying car in the X direction is (-20,100). The point cloud data within 10 meters on both sides of the Y axis of the flying car is retained, that is, the range in the Y direction is (-10,10). The range in the Z axis direction is (-2,5).

[0043] Step 3: Filter the point cloud data. For example, voxel filtering can be performed on the point cloud data to remove some dense and useless points in the original point cloud, while reducing the number of point clouds without affecting the geometric structure characteristics of the objects in the point cloud.

[0044] In the fourth step, the Patchwork++ algorithm is used to effectively segment the ground point cloud and the non-ground point cloud to adapt to different road scenes.

[0045] Step 5: In model training, the data collected by the LiDAR is distributed in a 1:1 ratio, with 50% as the training set and 50% as the validation set. Using the Pointpillars algorithm, the collected data is used to train and validate the point cloud data detection model.

[0046] Step 6: Acquire point cloud data of the flying car's surroundings. The present invention adopts a target detection method based on voxel features for point cloud feature extraction and detection. The point cloud data detection model performs target detection based on voxel features.

[0047] Step 3: Millimeter-wave radar raw data collection and processing: Collect the original image data of the flying car's surrounding environment to provide necessary data information for subsequent system data processing and decision-making systems.

[0048] Step 1: Turn on the millimeter-wave radar to collect a large amount of radar data of the surrounding environment.

[0049] Step 2: Cluster the targets of the millimeter-wave radar, use the DBSCAN algorithm to detect outliers, and then identify and eliminate the outliers simultaneously. For example, first select any reflection point in the millimeter-wave radar reflection signal as the initial value of the cluster center. Calculate the cluster center and reflection points Manhattan distance between and speed deviation , (1) (2) in Indicates distance, Indicates speed deviation represents the abscissa of the cluster center, represents the horizontal coordinate of the reflection point, Indicates the current speed. Indicates the speed at the next moment.

[0050] Will , The threshold values ​​are set , For comparison, if > ,and > , then it is determined that the two points do not belong to the same cluster.

[0051] Step 3: Screen effective targets. For example, first screen static targets. In the initial state, all targets collected by the millimeter-wave radar are regarded as independent clusters. . Compare all clusters pairwise, and you will get the two clusters with the smallest values. , , (3) (4) (5) in , Represents the two clusters with the smallest value, Indicates the minimum relative speed difference is the error when measuring speed within millimeter wave.

[0052] The specific process of screening false targets is as follows: Non-false targets need to meet three conditions at the same time: (6) (7) (8) The speed of the target at time t is , the angle is , the distance is , the speed of the target at time t+1 is , the angle is , the distance is The thresholds are , , .

[0053] For empty targets, the number of targets detected by the algorithm is less than the set number. For example, if the set number is 100, but the number of targets detected is 70, the others will output empty targets with a parameter of 0.

[0054] Step 4: Perform consistency judgment, such as checking the consistency of valid targets in two adjacent time periods based on the physical motion judgment criteria. Track valid targets based on the Kalman filter algorithm and update the target status in real time.

[0055] Step 4: Fig. 9 As shown, the camera detection results and the lidar detection results are fused: the two-dimensional image target information detected by vision and the target detection results obtained based on the lidar algorithm are fused.

[0056] In the first step, the 3D point cloud data of the LiDAR is displayed on a two-dimensional image. After the LiDAR and camera are synchronized and jointly calibrated in time and space, the point cloud corresponds one-to-one with the target on the image.

[0057] In the second step, the image detection data and the point cloud detection data are associated. For example, the intersection-over-union (IOU) matching strategy is used to effectively fuse the two independent detection results and filter out the point cloud targets outside the visual range and the point cloud targets with less overlap with the visual targets.

[0058] The target detection box IOU spatial distance is used as the weight, and its weight function is shown in the formula (9) in is the distance difference between the two target sets, is the overlap difference between the boxes of the two target sets. is the camera visual target set It is a collection of LiDAR point cloud targets.

[0059] Then the multi-sensor fusion adopts KM matching algorithm to get the optimal matching result.

[0060] Step 5: Fig.10 As shown, the millimeter-wave radar data is used to correct the confidence of the detection result: the millimeter-wave radar data is used to correct the confidence of the target output by the fusion of camera and lidar.

[0061] Step 1: Use the angle index to filter the fused target result and the filtered millimeter-wave radar data target detection result. If the angle difference between the two targets is used as the weight, the weight function is as shown in the formula (10) in is the distance difference between the two target sets, is the overlap difference between the boxes of the two target sets. It means that after the lidar and vision are matched, a fusion target set and a single sensor target set that is not matched will be obtained. Represents the target set obtained after filtering the millimeter-wave radar data.

[0062] Then the KM matching algorithm is used to obtain the optimal matching result.

[0063] Step 2: Use the weighted average method to correct the target confidence. For example, since the millimeter-wave radar only has target distance and speed information but no target category information, the millimeter-wave radar target category confidence is set to 1 and the visual target confidence is modified using the weighted average method. The formula is as follows: (11) In the formula is the visual target category confidence, is the point cloud target category confidence, , and are the weights of their respective target categories, is the confidence of the visual target category after fusion.

[0064] Step 3: Adjust the weight according to the credibility. For example, if the original confidence image target is high, its credibility is high, and the image target weight can be retained or increased to reduce the possible false detection caused by the point cloud classification algorithm and millimeter wave radar target. For the low confidence image target added later, the target confidence can be improved through the point cloud algorithm and millimeter wave radar data. (12) Step 4: Output the results. The information fused from the camera, lidar and millimeter-wave radar is output to the control module to control the flying car to switch modes.

[0065] Step 6: Figure 8 As shown, the mode is switched based on the data fused by the camera, lidar, and millimeter-wave radar.

[0066] Step 1: Determine whether the flying car is in flight mode or land mode.

[0067] Step 2: If the vehicle is in land mode, the result of multi-sensor fusion is used to determine whether the mode can be switched. If there are no obstacles within the preset distance around the flying car and the driver activates the mode switching control module, the mode can be switched and a signal to allow the mode switch can be issued. If only one of the above conditions is met, the mode cannot be switched.

[0068] Step 3: If the mode conversion can be performed, determine which mode conversion method to use based on user needs. If there are no obstacles in the preset road ahead of the flying car, the flying car will take off by rolling takeoff, open the retractable wings, and use the foldable ducted fan as an additional thrust device to enable the flying car to reach the speed required for rolling takeoff. If there are obstacles in the preset road ahead and the driver still wants to take off, determine whether there are obstacles around the flying car. If the take-off conditions are met, use the foldable ducted fan for vertical takeoff. Otherwise, the flying car cannot perform the takeoff mode conversion and reminds the driver.

[0069] Step 4: If the flying car is in flight mode and the driver performs a landing operation, the multi-sensor fusion is used to determine whether the surrounding environment meets the requirements for mode conversion. If the judgment result is that there is no obstruction, a signal to allow mode conversion is issued to perform vertical landing. If the flying car first flies to a safe distance above the landing area according to the driver's instructions, it will perform vertical landing when there are no obstacles around the flying car according to the results of multi-sensor fusion.

[0070] The following are device embodiments of the present invention, which can be used to perform method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0071] In yet another embodiment of the present invention, a flying car mode conversion system is provided, which can be used to implement the above-mentioned flying car mode conversion method. Specifically, the flying car mode conversion system includes a target detection module, a fusion module, a correction module and a judgment module.

[0072] Among them, the target detection module is used to obtain image data, point cloud data and radar data of the flying car's surrounding environment, and perform target detection respectively.

[0073] The fusion module is used to fuse the target detection results of image data and point cloud data.

[0074] The correction module is used to correct the confidence of the fusion result using the target detection result of the radar data.

[0075] The judgment module is used to judge whether the surrounding environment meets the requirements of mode conversion according to the correction result. If the judgment result is that there is no obstruction, a signal allowing mode conversion is issued; if the judgment result is that there is obstruction, a signal prohibiting mode conversion is issued.

[0076] In yet another embodiment of the present invention, a terminal device is provided. The terminal device includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions; the processor described in the embodiment of the present invention can be used for the operation of the flying car mode conversion method, including: S1, acquiring image data, point cloud data and radar data of the surrounding environment of the flying car, and performing target detection respectively; S2, fusing the target detection results of the image data and the point cloud data; S3, using the target detection result of the radar data to correct the confidence of the fusion result; S4, judging whether the surrounding environment meets the requirements of the mode conversion according to the correction result, if the judgment result is that there is no obstruction, a signal allowing the mode conversion is issued, and if the judgment result is that there is obstruction, a signal prohibiting the mode conversion is issued.

[0077] In another embodiment, the present invention further provides a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.

[0078] The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the flying car mode conversion method in the above embodiment; the processor loads one or more instructions in the computer-readable storage medium and executes the following steps: S1, acquiring image data, point cloud data and radar data of the flying car's surrounding environment, and performing target detection respectively; S2, fusing the target detection results of the image data and the point cloud data; S3, using the target detection result of the radar data to correct the confidence of the fusion result; S4, judging whether the surrounding environment meets the requirements of the mode conversion according to the correction result, if the judgment result is that there is no obstruction, a signal allowing the mode conversion is issued, and if the judgment result is that there is obstruction, a signal prohibiting the mode conversion is issued.

[0079] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0080] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0083] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0084] It should be understood that the above description is for illustration and not for limitation. Many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of this patent should not be determined with reference to the above description, but rather with reference to the preceding claims and the full scope of equivalents to which such claims are entitled. For the purpose of comprehensiveness, all articles and references, including disclosures of patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the preceding claims is not intended to be a waiver of such subject matter, nor should it be considered that the applicant has not considered such subject matter to be part of the disclosed inventive subject matter.

Claims

1. A flying car mode conversion method, characterized in that: The process includes: S1, obtains image data, point cloud data and radar data of the flying car's surrounding environment, and performs target detection respectively; S2, fuses the target detection results of image data and point cloud data; S3, using the target detection result of radar data to correct the confidence of the fusion result; S4, judging whether the surrounding environment meets the requirements of mode switching according to the correction result, if the judgment result is that there is no obstruction, a signal allowing mode switching is issued, if the judgment result is that there is obstruction, a signal prohibiting mode switching is issued.

2. The flying car mode conversion method according to claim 1, characterized in that: In S1, the YOLOV8 model is used to detect targets on image data. The process of establishing the YOLOV8 model is as follows: collect image data of the surrounding environment; annotate the collected original image information; use random rotation, color transformation, random noise and other methods to enhance the data; train the YOLOV8 model used to detect image targets to obtain the optimal model.

3. The flying car mode conversion method according to claim 1, characterized in that: In S1, a point cloud data detection model is used to detect targets on point cloud data based on voxel features. The process of establishing the point cloud data detection model is as follows: collect point cloud data of the surrounding environment; divide the point cloud data into effective areas; perform voxel filtering on the point cloud and downsample the point cloud; use the Patchwork++ algorithm to segment the point cloud; and use the Pointpillars algorithm to train and verify the model using the collected point cloud data.

4. The flying car mode conversion method according to claim 1, characterized in that: In S1, the process of target detection on radar data is as follows: collecting radar data of the surrounding environment; clustering the radar data, using the DBSCAN algorithm to detect outliers, and then distinguishing the outliers and eliminating them simultaneously; screening valid targets, preliminarily screening out stationary targets, false targets and empty targets; and performing consistency check and tracking on valid targets.

5. The flying car mode conversion method according to claim 1, characterized in that: The specific process of S2 is: display the point cloud data on a two-dimensional image, and correspond the targets on the point cloud data and the image data one by one; use the evaluation index of the target distance and the intersection and union ratio of the point cloud image target to associate the data to obtain the possible matching weights of each target, arrange them, and then use the KM algorithm to obtain the optimal matching result, and output the result and confidence.

6. The flying car mode conversion method according to claim 1, characterized in that: The specific process of S3 is as follows: the fused target results and radar data target detection results are screened using angle indicators; the distance is used as the difference judgment indicator for the target matching after screening to obtain the possible matching weight of each target, and the order is arranged according to the weight; the KM matching algorithm is used to obtain the optimal match between the fused target and the millimeter-wave radar target; and the weighted average method is used to correct the confidence of the matched images, point clouds and radar targets.

7. The flying car mode conversion method according to claim 1, characterized in that: The specific process of S4 is as follows: determining whether the flying car is in flight mode or land mode; if in land mode, determining whether the mode conversion requirements are met based on the correction result; when there are no obstacles within a preset distance around the flying car and the driver initiates the control command for mode conversion, the mode conversion is performed and a taxiing takeoff is performed; if the judgment result is that there is an obstruction within the preset road, determining whether the surrounding environment of the flying car meets the conditions for vertical takeoff, if so, a vertical takeoff is performed; if neither is satisfied, a signal prohibiting mode conversion is issued; if the flying car is in flight mode and the driver performs a landing operation, the landing is performed based on the correction result when there are no obstacles within a preset distance around the flying car.

8. A flying car mode conversion system, characterized in that: include: The target detection module is used to obtain image data, point cloud data and radar data of the flying car's surrounding environment and perform target detection respectively; A fusion module is used to fuse the target detection results of image data and point cloud data; A correction module is used to correct the confidence of the fusion result using the target detection result of the radar data; The judgment module is used to judge whether the surrounding environment meets the requirements of mode conversion according to the correction result. If the judgment result is that there is no obstruction, a signal allowing mode conversion is issued; if the judgment result is that there is obstruction, a signal prohibiting mode conversion is issued.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the flying car mode conversion method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the flying car mode conversion method as described in any one of claims 1 to 7 are implemented.

Images