Aircraft three-dimensional obstacle avoidance perception method and system based on radar vision fusion

Abstract

This invention provides a method and system for three-dimensional obstacle avoidance perception of aircraft based on radar-vision fusion, belonging to the field of autonomous obstacle avoidance control technology for aircraft. It solves the limitations of existing obstacle avoidance schemes, such as low accuracy, slow response, and high false negative rate. The method first acquires guidance radar data and path image data, performs time synchronization and alignment, and then uses a pre-constructed lightweight feature fusion network to extract and cross-correlate features, resulting in a multimodal feature map. Based on the multimodal feature map and the aircraft's current flight status data, a dynamic spatial grid containing predicted flight paths and detected obstacles is generated. A preset loss function is used to optimize the accuracy of the dynamic spatial grid, and based on the optimized dynamic spatial grid, a decision result containing the aircraft's maneuver commands is output. This invention significantly improves pose perception accuracy, obstacle detection rate, real-time performance, and stability, ensuring effective three-dimensional obstacle avoidance perception for aircraft.

Description

Technical Field

[0001] This invention relates to the field of autonomous obstacle avoidance control technology for aircraft, specifically to a three-dimensional obstacle avoidance perception method and system for aircraft based on radar-visual fusion. Background Technology

[0002] In the autonomous navigation and mission execution of low-altitude aircraft, three-dimensional environmental perception capability is a crucial foundation for ensuring their safe and efficient operation. Especially in complex urban or near-ground environments, aircraft need to avoid various static and dynamic obstacles, such as tall buildings, power transmission towers, communication facilities, and other aerial targets, while maneuvering at high speeds. These scenarios place extremely high demands on the accuracy, real-time performance, and robustness of the perception system. Current mainstream three-dimensional environmental perception methods mostly rely on pure vision or lidar solutions, which perform well in some static or low-speed scenarios, but still face significant challenges under high-speed, high-maneuver conditions.

[0003] Existing vision-based 3D reconstruction methods typically rely on visual odometry for pose estimation. However, during high-dynamic maneuvers such as sharp turns, dives, or rapid changes of direction, pose estimation errors accumulate rapidly, causing significant shifts in the constructed environmental model. This leads to obstacle localization failure and makes it difficult to support reliable obstacle avoidance decisions. Furthermore, although some tasks have external ranging information sources, such as distance data from remote guidance systems, existing purely visual perception frameworks generally do not effectively integrate this prior information. This results in insufficient depth estimation accuracy in mid- to long-range regions, making it difficult to meet collision avoidance requirements in high-risk areas.

[0004] Traditional 3D perception systems often use fixed-range voxel networks or feature maps for environmental modeling, resulting in a uniform distribution of computational resources without dynamically focusing on critical areas ahead of the flight path. This static modeling strategy not only leads to a large amount of redundant computation but also causes delays in environmental updates in high-risk areas, making it impossible to complete obstacle detection and response within a limited time window. Furthermore, existing perception models often employ globally consistent loss functions during training or optimization, failing to assign higher priority to specific obstacles with high threats in the mission scenario (such as slender towers or dense building clusters), resulting in a high rate of missed detections of critical obstacles and seriously threatening the flight safety of the aircraft.

[0005] It is evident that existing 3D environmental perception technologies still have significant shortcomings in terms of dynamic adaptability, multi-source information fusion, scene focusing capabilities, and obstacle priority modeling, making it difficult to meet the comprehensive requirements of aircraft for accurate, real-time, and reliable obstacle avoidance perception in high-speed and complex environments. Therefore, there is an urgent need for a novel 3D obstacle avoidance perception method that can combine external ranging and guidance information, possess dynamic modeling capabilities, and enhance the perception of key obstacles, in order to improve the autonomous and safe flight capabilities of aircraft in complex low-altitude environments. Summary of the Invention

[0006] The purpose of this invention is to address the limitations of existing 3D obstacle avoidance perception methods in high-speed maneuvering scenarios, such as low accuracy, slow response, and high false negative rates due to large pose errors, lack of external ranging information fusion, static redundancy in environmental modeling, and insufficient perception of key obstacles. Therefore, this invention proposes a radar-visual fusion-based 3D obstacle avoidance perception method and system for aircraft. This invention utilizes radar-guided distance information and a stabilized camera's perspective to form a multi-source fusion, dynamic focusing, and risk-priority 3D environmental perception scheme, enabling 3D obstacle avoidance perception for aircraft. This invention significantly improves pose perception accuracy, obstacle detection rate, real-time performance, and stability, ensuring the overall superior performance of 3D obstacle avoidance perception for aircraft.

[0007] The present invention employs the following technical solutions to achieve its objective:

[0008] A three-dimensional obstacle avoidance perception method for aircraft based on radar-vision fusion includes the following steps:

[0009] S1. During the flight of the aircraft to the target, acquire guidance radar data from the external radar system and path image data from the guidance camera on the aircraft, and perform time synchronization and alignment.

[0010] S2. For the synchronized guidance radar data and path image data, a pre-built lightweight feature fusion network is used to extract and cross-correlate features, and the fusion results in a multimodal feature map.

[0011] S3. Based on the multimodal feature map and the current flight status data of the aircraft, generate a dynamic spatial grid that includes predicted flight path and detected obstacles;

[0012] S4. Optimize the accuracy of the dynamic space grid using a preset loss function. Based on the optimized dynamic space grid, output the decision results containing the aircraft's maneuver commands to complete the aircraft's three-dimensional obstacle avoidance perception.

[0013] Preferably, in step S1, the relative angle between the guidance camera and the aircraft is acquired simultaneously to characterize the pointing deviation of the guidance camera during the aircraft's flight to the target; at the same time, the aircraft's attitude information is acquired through the aircraft's IMU system for attitude correction when the aircraft performs maneuvers.

[0014] When performing time synchronization alignment, the timestamp of the guidance radar data is used as a reference, and the delay of the path image data and the aircraft attitude information is compensated by interpolation. Then, the geodetic coordinate system pose of the guidance camera is calculated. This geodetic coordinate system pose is used to assist in the generation of the dynamic space grid in step S3.

[0015] Specifically, when calculating the geodetic coordinate system pose of the guidance camera, the proportional guidance method is used to determine the predicted flight path of the aircraft, and the relative angle between the guidance camera and the aircraft, as well as the aircraft attitude information, are used for direct calculation.

[0016] When performing time synchronization alignment, for data frames in the guidance radar data whose jump distance exceeds the preset threshold, their timestamps are still used as the reference, but low confidence markers are added, and their corresponding weights are reduced when feature fusion is performed in step S2.

[0017] Preferably, in step S2, the distance information in the guidance radar data and the pixel information in the path image data are correlated according to the synchronized guidance radar data and path image data, and used as the input of the lightweight feature fusion network. The lightweight feature fusion network adopts the ShuffleNetV2 network, which uses the distance information in the guidance radar data as the depth prior data, strengthens the area in the path image data that matches the guidance radar data through the spatial attention mechanism, and suppresses the area in the path image data that is unrelated to the flight path.

[0018] Specifically, in the ShuffleNetV2 network, the input radar feature map is obtained by converting the guidance radar data, while the visual feature map is extracted from the path image data. For the cross function of the spatial attention mechanism, it calculates the similarity between the visual features and the radar features, with the radar features as the dominant feature, and dynamically allocates the feature weights during fusion. The higher the confidence of the radar feature map, the higher its feature weight. If the confidence of the radar feature map is lower than a preset threshold, then the visual features are used as the dominant feature.

[0019] The fused multimodal feature map shows the obstacle features on the flight path of the aircraft to the target, thus representing the pixel corresponding to the obstacle in the path image data and the radar distance corresponding to that pixel.

[0020] Preferably, in step S3, the current flight status data of the aircraft includes the geodetic coordinate system pose of its guidance camera and the current velocity of the aircraft. Based on the multimodal feature map and flight status data, a three-dimensional space is divided within the same or different preset distance ranges corresponding to the forward, backward, left and right and near-ground sides of the predicted flight path, with the current position of the aircraft as the origin, thereby generating a dynamic spatial grid. The dynamic spatial grid adopts an adaptive resolution adjustment strategy. For the region that is far from the predicted flight path within a first preset distance range, a dense coding method is applied. For the region that is far from the predicted flight path within a second preset distance range, a sparse sampling method is applied. The maximum value of the first preset distance range is less than the minimum value of the second preset distance range.

[0021] Preferably, during the flight of the aircraft to the target, the dynamic space grid is incrementally updated according to the output of each frame of the multimodal feature map, adding the newly appearing grid in the three-dimensional space of the predicted flight path corresponding to each frame output, and deleting the area outside the three-dimensional space that has been flown through after the displacement of the aircraft.

[0022] For regions that are a third preset distance from the predicted flight path, full-resolution modeling is applied; the set corresponding to the third preset distance is a subset of the set corresponding to the first preset distance, and the minimum value of the third preset distance is equal to the minimum value of the first preset distance; when archiving any data frame, for regions outside the third preset distance, an octree structure compression method is used for data storage.

[0023] Preferably, in step S4, the preset loss function is a weighted loss function, including the reprojection error loss of key obstacles, the general loss of the dynamic spatial grid, the background loss, the radar constraint loss, and the path consistency loss.

[0024] As the aircraft gradually flies towards the target, when the distance between the aircraft and the target is less than a preset distance threshold, the weight of the radar constraint loss is increased by a preset value based on the initial value, and the sum of the remaining weights is adaptively reduced by a preset value; the path consistency loss is achieved through neighborhood convolution of a preset size.

[0025] Preferably, based on the optimized dynamic spatial grid, a dynamic threshold decision-making method is adopted. When an obstacle feature appears in the predicted area of ​​the dynamic spatial grid and the IoU value corresponding to the obstacle feature reaches the preset warning threshold, a multi-level warning decision is triggered. The multi-level warning decision classifies the obstacle risk level into high risk, medium risk and low risk according to the radar distance corresponding to different spatial grids that reach the preset warning threshold.

[0026] For high-risk levels, output hard evasion maneuver commands for the aircraft that include lateral acceleration constraints;

[0027] For medium-risk levels, the output includes aircraft attitude change commands that include pitch angle adjustments;

[0028] For low-risk levels, only proportional guidance is used to update the flight path planning and the predicted flight path of the aircraft.

[0029] This invention also provides a three-dimensional obstacle avoidance perception system for implementing the above-mentioned three-dimensional obstacle avoidance perception method for aircraft. The system includes the following functional modules:

[0030] The data synchronization module is used to acquire guidance radar data from external radar systems and path image data from guidance cameras on the aircraft, and to perform time synchronization and alignment.

[0031] The feature fusion module is used to extract and cross-correlate features from guidance radar data and path image data using a pre-built lightweight feature fusion network, and obtain a multimodal feature map after fusion.

[0032] The mesh generation module is used to generate a dynamic spatial mesh containing predicted flight paths and detected obstacles based on multimodal feature maps and the current flight status data of the aircraft.

[0033] The loss optimization module is used to optimize the accuracy of the dynamic space grid using a preset loss function. Based on the optimized dynamic space grid, it outputs decision results containing aircraft maneuver commands.

[0034] In summary, due to the adoption of this technical solution, the beneficial effects of this invention are as follows:

[0035] This invention significantly improves the accuracy of three-dimensional environmental perception of aircraft during high-speed maneuvers by fusing external ranging information with a stable imaging perspective. Compared to traditional methods that rely on visual odometry, this invention employs a direct pose calculation mechanism, effectively suppressing pose drift caused by violent motion and greatly improving the accuracy of obstacle spatial positioning.

[0036] This invention introduces a dynamically adjustable 3D modeling range, concentrating computational resources on key areas ahead of the flight path. This avoids redundant processing caused by fixed grids, significantly reducing processing latency while ensuring perception coverage, and meeting the stringent real-time requirements of high frame rate decision-making. Furthermore, a priority perception mechanism is designed for specific obstacles in the environment that have a significant impact on flight safety. This mechanism strengthens the feature representation of these obstacles during model training and inference, significantly reducing the probability of missing detection of critical obstacles.

[0037] Furthermore, this invention constructs an adaptive fusion strategy for multi-source data, which can dynamically adjust the weight of external ranging information in feature fusion based on its confidence level, maintaining stable depth estimation performance even in complex electromagnetic or interference environments. The overall solution achieves significant improvements in sensing accuracy, response speed, detection reliability, and environmental adaptability, providing strong support for the safe autonomous flight of aircraft in complex low-altitude scenarios. Attached Figure Description

[0038] The present invention further illustrates its embodiments and technical solutions in detail with reference to the following figures, specifically including 5 figures as follows:

[0039] Figure 1 This is a schematic diagram illustrating the overall process of the three-dimensional obstacle avoidance perception method for aircraft of the present invention;

[0040] Figure 2 This is a schematic diagram of the module hierarchy and core data flow of the system of the present invention;

[0041] Figure 3 This is a timing diagram illustrating data time synchronization and alignment in the system of the present invention;

[0042] Figure 4 This is a schematic diagram of the dynamic spatial grid generation mechanism in the system of the present invention;

[0043] Figure 5 This is a schematic diagram of the optimization strategy of the loss optimization module in the system of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0045] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0046] Example 1

[0047] A radar-vision fusion-based three-dimensional obstacle avoidance perception method for aircraft. Figure 1 This document provides a brief overview of the overall process of the method, which can be viewed concurrently. The key steps of the method can be summarized as follows:

[0048] S1. During the flight of the aircraft to the target, acquire guidance radar data from the external radar system and path image data from the guidance camera on the aircraft, and perform time synchronization and alignment.

[0049] S2. For the synchronized guidance radar data and path image data, a pre-built lightweight feature fusion network is used to extract and cross-correlate features, and the fusion results in a multimodal feature map.

[0050] S3. Based on the multimodal feature map and the current flight status data of the aircraft, generate a dynamic spatial grid that includes predicted flight path and detected obstacles;

[0051] S4. Optimize the accuracy of the dynamic space grid using a preset loss function. Based on the optimized dynamic space grid, output the decision results containing the aircraft's maneuver commands to complete the aircraft's three-dimensional obstacle avoidance perception.

[0052] The aforementioned method aims to improve the accuracy and robustness of obstacle detection in complex flight environments. By collaboratively processing guidance data provided by external radar systems and path visual information captured by the aircraft's own guidance cameras, it achieves temporal alignment and feature-level deep fusion of multi-source data. Furthermore, a lightweight network model is used to efficiently extract and cross-correlate features from multimodal inputs, constructing a fusion feature representation with spatial semantic capabilities. Based on this, a spatial grid model of the dynamic environment is generated by combining real-time aircraft status information, which can predict flight trajectories and identify potential obstacle distributions. Finally, through targeted accuracy optimization, a perception output containing obstacle avoidance decisions and flight maneuver commands is generated, forming a closed-loop intelligent obstacle avoidance perception system that provides reliable support for autonomous aircraft navigation.

[0053] This embodiment will describe in detail the specific or preferred contents of each step in the order of the steps described above.

[0054] In step S1, the preferred distance range for the acquired guidance radar data is between 0 and 70 meters from the aircraft. This radial distance information is updated in real time to meet the requirements of the aircraft's flight to the target under specific low-altitude conditions. The path image data from the guidance camera on the aircraft has a resolution of 640×480, is in RGB image format, and has a frame rate of 30FPS, corresponding to the field of view of the area locked by the guidance radar data.

[0055] In step S1, the relative angle between the guidance camera and the aircraft is acquired simultaneously to characterize the pointing deviation of the guidance camera during the aircraft's flight to the target; at the same time, the aircraft's attitude information is acquired through the aircraft's IMU system, focusing on its Euler angles, which are used for attitude correction when the aircraft performs maneuvers.

[0056] During time synchronization alignment, the timestamp of the guidance radar data is used as a reference. Cubic spline interpolation is used to compensate for the delays of the path image data and the aircraft attitude information, thereby ensuring the spatial consistency of multi-source data at the same moment during guidance. The guidance radar data needs to be filtered during time synchronization. A sliding window filtering method can be used to smooth it, suppress high-frequency noise, and stabilize the range information error it represents to a certain extent beforehand. Then, the geodetic coordinate system pose of the guidance camera is calculated. This geodetic coordinate system pose is used to assist in the generation of the dynamic spatial grid in step S3.

[0057] In this embodiment, when calculating the geodetic coordinate system pose of the guidance camera, the proportional guidance method is used to determine the predicted flight path of the aircraft. The relative angle between the guidance camera and the aircraft, along with the aircraft's attitude information, is then used for direct calculation. A key feature of this method is that it does not require the use of a visual odometry system. The corresponding formula is as follows:

[0058]

[0059] In the formula, To guide the camera's geodetic pose; The attitude rotation matrix for guiding the aircraft is obtained based on the aircraft's attitude information; The relative rotation matrix of the guidance camera is obtained based on the relative angle between the guidance camera and the aircraft, and can be implemented using quaternion transformation.

[0060] As a preferred embodiment, when performing time synchronization alignment, for data frames in the guidance radar data with a jump distance exceeding 2m, they may be subject to electronic interference, but they are still normally based on their timestamps, with only an additional low-confidence marker added, and their corresponding weights are reduced during feature fusion in step S2.

[0061] In step S2, based on the synchronized guidance radar data and path image data, the distance information in the guidance radar data is correlated with the pixel information in the path image data, and used as input to the lightweight feature fusion network. In this embodiment, the lightweight feature fusion network uses the ShuffleNetV2 network as the backbone network, which has 1.4M parameters and can replace traditional heavy networks. It can meet the real-time requirements of the aforementioned path image data at 30FPS on Jetson AGXOrin.

[0062] The ShuffleNetV2 network uses distance information from the guidance radar data as depth prior data. It enhances the regions in the path image data that match the guidance radar data through a spatial attention mechanism. For example, when both can represent a tower 70m away from the aircraft, there are corresponding image pixels and distance information. At the same time, it suppresses regions in the path image data that are not related to the flight path, such as the background sky and ground outside the flight path.

[0063] In this embodiment, the fusion calculation formula of the ShuffleNetV2 network is as follows:

[0064]

[0065] In the formula, The fused multimodal feature map contains visual pixel information and radar distance information; This is a visual feature map extracted from path image data by the ShuffleNetV2 network, involving features such as edges and textures; The radar feature map obtained from the guidance radar data is normalized to 0-1 within the range of 0-70m. Use the Softmax activation function; The feature dimension of the visual feature map and the radar feature map is the length of their feature vectors.

[0066] This is the cross function corresponding to the spatial attention mechanism. It calculates the similarity between visual features and radar features, with radar features as the dominant feature, and dynamically allocates feature weights during fusion. The higher the confidence level of the radar feature map, the higher its feature weight. If the confidence level of the radar feature map is lower than a preset threshold, then visual features are used as the dominant feature.

[0067] The fusion calculation formula calculates the dot product of visual features and radar features and divides by . This approach achieves scale normalization of similarity, then converts it into attention weights via a Softmax function, thereby dynamically allocating fusion weights for different modal features. It employs a spatial attention mechanism, prioritizing radar features and dynamically adjusting the fusion ratio based on their confidence level: when radar features have high confidence, they are given higher weights; conversely, visual features take the lead. Overall, this fusion strategy achieves adaptive weighted fusion of multimodal features, improving perceptual robustness and accuracy in complex environments.

[0068] The fused multimodal feature map shows the obstacle features on the flight path of the aircraft to the target, thus representing the pixel corresponding to the obstacle in the path image data and the radar distance corresponding to that pixel.

[0069] In step S3, the current flight status data of the aircraft includes the geodetic coordinate system pose of its guidance camera and the current velocity of the aircraft. By combining the multimodal feature map, a dynamic spatial grid can be generated and output. The geodetic coordinate system pose of the guidance camera can accurately represent the global position here, which helps to locate the absolute coordinates of the aircraft in the environment. The current velocity of the aircraft is used to calculate its motion trend and predict the trajectory extension direction by combining it with the proportional guidance method.

[0070] In this embodiment, based on the multimodal feature map and flight status data, a three-dimensional space is divided within the distance range corresponding to the predicted flight path: 70m forward, 10m backward, and 20m to the left and right, with the current position of the aircraft as the origin. The forward direction is the extension direction of the predicted trajectory based on proportional guidance, 10m backward is required to retain nearby obstacles, and 20m to the left and right covers the aircraft's evasive maneuver range. At the same time, a range of -5 to 20m is also planned on the z-axis of the geodetic coordinate system as the key obstacle area from the ground to the air, to jointly generate a dynamic spatial grid.

[0071] The dynamic spatial grid adopts an adaptive resolution adjustment strategy. For areas within 0-30m of the predicted flight path, dense coding is applied to improve spatial resolution and accurately capture details and potential obstacles. For areas within 31-70m of the predicted flight path, sparse sampling is applied to reduce resolution to save computing resources and expand coverage. In other words, 30m is used as the threshold for dense and sparse resolution adjustment in this embodiment.

[0072] As the basic unit of the mesh, the resolution of a voxel determines the fineness of the spatial division. Dense coding contains more voxels per unit volume, achieving high-precision detection, while sparse sampling contains fewer voxels per unit volume, optimizing processing efficiency. This strategy ensures a balance between accuracy and performance in complex environments, supporting efficient navigation and obstacle avoidance for aircraft. In this embodiment, the voxel resolution is 0.3m for the 0-30m region, with 111 voxels per cubic meter under dense coding; for the 31-70m region, the voxel resolution is 0.5m, with 80 voxels per cubic meter under sparse sampling.

[0073] In this embodiment, during the flight of the aircraft to the target, the dynamic spatial grid is incrementally updated based on the output of each frame of the multimodal feature map. For the three-dimensional space of the predicted flight path corresponding to each frame output, the three-dimensional space at this time is the range from the current position -10m to the current position +70m in the flight path direction. As the aircraft moves, only the new grid appearing in front of it is added, and the area outside the three-dimensional space that has been flown through after the aircraft's displacement is deleted, that is, the area flown through before the current position -10m. The dynamic spatial grid is thus continuously updated to maintain a real-time state.

[0074] As a preferred embodiment, for areas within 0-5m of the predicted flight path, full-resolution modeling is applied, and this area is considered an absolutely high-risk area. When archiving any data frame, for areas outside the 0-5m range, an octree compression method is used for data storage. This allows large blocks of space to be represented with fewer nodes in open or relatively static areas, thus saving storage space. Octree compression can significantly reduce the storage volume of 3D environment data, especially when most external areas are open or repetitive. Compared to storing the entire space using a voxel grid, an octree only needs to store the meaningful parts, achieving efficient representation of sparse data.

[0075] In step S4, the preset loss function is a weighted loss function, and its function is as follows:

[0076]

[0077] In the formula, to Preset weights for each loss term; For reprojection error loss of critical obstacles; This is the typical loss for dynamic spatial grids; Background loss; For radar constraint loss; This represents the path consistency loss.

[0078] As the aircraft gradually flies towards the target, when the distance between the aircraft and the target is less than 30m, radar constraint loss occurs. weight Increase the preset value based on the initial value, and adaptively decrease the preset value of the sum of the remaining weights; path consistency loss. This is achieved through 3×3×3 neighborhood convolution.

[0079] In this embodiment, the reprojection error loss of critical obstacles is a loss item for the spatial position prediction accuracy of high-risk, non-circumventable or difficult-to-bypass, and dynamically moving obstacles, such as fixed buildings like power transmission towers and other aircraft. The reprojection error represents the position deviation when projecting a three-dimensional point onto a two-dimensional image plane. This loss item measures the projection error of critical obstacles on the image plane and plays a role in avoiding missed detections or misjudgments.

[0080] The normal loss of the dynamic space mesh is a general normal or basic loss, which is only used to measure the overall reconstruction quality of the entire dynamic space mesh and can characterize some non-critical obstacle objects, that is, obstacles that are far away from the flight path and can be easily avoided or have no impact. The background loss is used to constrain the modeling behavior of non-obstacle areas to prevent the free space from being mistakenly marked as obstacles.

[0081] Radar constraint loss is used to measure the inconsistency between the prediction results of the dynamic space grid and the actual detection results of the guiding radar data. If the radar does not detect an obstacle in a certain area, but the grid generation marks its presence or proportion, a penalty is incurred. When the aircraft approaches the target, the weight of the radar loss increases, indicating that more attention should be paid to the radar data, because radar information at close range is more reliable.

[0082] Path consistency loss ensures that the generated paths or decisions are continuous, smooth, and conform to physical laws in time. When neighborhood convolution implements this loss term, it compares the consistency between the current state and the historical state within a local spatial region.

[0083] In this embodiment, based on the optimized dynamic spatial grid, a dynamic threshold decision-making method is adopted. When an obstacle feature appears in the predicted area of ​​the dynamic spatial grid and the IoU value corresponding to the obstacle feature reaches the warning threshold of 0.8, a multi-level warning decision is triggered. The multi-level warning decision classifies the obstacle risk level into high risk, medium risk and low risk according to the radar distance corresponding to different spatial grids that have reached the warning threshold.

[0084] For high-risk levels, the radar distance is less than 10m, and the output includes hard evasion maneuver commands for the aircraft that include lateral acceleration constraints.

[0085] For medium-risk levels, the radar range is 10-30m, and the output includes aircraft attitude change commands that include pitch angle adjustment;

[0086] For low-risk levels, with a radar range of 30-70m, only the proportional guidance method is used to update the flight path planning and the predicted flight path of the aircraft.

[0087] Example 2

[0088] Based on Example 1, this example provides a radar-vision fusion-based three-dimensional obstacle avoidance perception system for aircraft, which can implement the three-dimensional obstacle avoidance perception method for aircraft in Example 1. In this example, the system includes the following functional modules:

[0089] The data synchronization module is used to acquire guidance radar data from external radar systems and path image data from guidance cameras on the aircraft, and to perform time synchronization and alignment.

[0090] The feature fusion module is used to extract and cross-correlate features from guidance radar data and path image data using a pre-built lightweight feature fusion network, and obtain a multimodal feature map after fusion.

[0091] The mesh generation module is used to generate a dynamic spatial mesh containing predicted flight paths and detected obstacles based on multimodal feature maps and the current flight status data of the aircraft.

[0092] The loss optimization module is used to optimize the accuracy of the dynamic space grid using a preset loss function. Based on the optimized dynamic space grid, it outputs decision results containing aircraft maneuver commands.

[0093] Figure 2 The system's module hierarchy and core data flow are shown. The input layer relies on three key data sources: guidance radar data, path image data, and flight status data. Guidance radar data provides information such as target distance and speed, path image data provides visual features, and flight status data is the aircraft's own IMU attitude, thus reflecting its own motion state. Together, these three constitute the foundation of multimodal perception.

[0094] The data synchronization module is the system's data entry point; please refer to [link / reference]. Figure 3 The diagram illustrates that the system receives and preprocesses three types of data inputs and solves the time synchronization problem of multi-source data; the feature fusion module extracts and cross-correlates features from the synchronized image and radar data, combining the advantages of visual details and radar distance; the mesh generation module constructs a dynamic spatial mesh based on the fused multimodal feature map, which can quantify the distribution of obstacles on the flight path; the loss optimization module optimizes the mesh accuracy in a targeted manner through its loss function, and finally outputs the decision result.

[0095] The system output layer serves the aircraft's obstacle avoidance execution, and can output information including obstacle coordinates, collision risk level, and avoidance maneuver commands, forming a closed loop of perception-decision-control.

[0096] In the data synchronization module, the three types of data from the radar system, guidance camera, and IMU system have different delays. Path image data has the longest delay, while flight status data has the shortest delay. This is an unavoidable hardware response characteristic, thus requiring time alignment through synchronization processing logic. The data synchronization module uses a cubic spline interpolation algorithm to backtrack the delayed data to the same time point, solving the sensor asynchronous problem. This module also quickly calculates the aircraft camera attitude based on the proportional guidance method, and the calculation process does not rely on time-consuming visual odometry. The guidance radar data is also smoothed for noise through a 5-frame moving average in this module, improving the stability of distance measurement. The synchronized data can also be divided into corrected images, camera attitude, and filtered radar distance, and then uniformly input into the feature fusion module to ensure the spatiotemporal consistency of subsequent processing.

[0097] After the feature fusion module completes feature extraction and cross-correlation to obtain a multimodal feature map, it is handed over to the mesh generation module to generate a dynamic spatial mesh. For example... Figure 4 As shown, the mesh generation module starts from the current position of the aircraft, first uses the proportional guidance method to determine the predicted flight path, and then defines the spatial boundary of the mesh coverage. That is, it divides the three-dimensional space within the distance range corresponding to 70m forward, 10m backward, and 20m left and right of the predicted flight path, and also plans the range of -5 to 20m on the z-axis of the geodetic coordinate system, thereby focusing on the core area around the flight path.

[0098] The mesh generation module dynamically adjusts its mesh precision based on distance. A dense 0.3m encoding is applied in the 0-30m range to ensure high precision at close range, while a sparse 0.5m sampling is applied in the 31-70m range, reducing precision at longer distances while maintaining efficiency, thus achieving a balance between accuracy and computational cost. For voxel optimization, high-risk areas near obstacles retain their full resolution, while other areas are stored using octree compression to merge empty voxels, reducing storage and computational overhead. The incremental update mechanism of the mesh generation module is triggered by aircraft displacement, eliminating the need for full reconstruction and reducing computational cost. For the embedded platform used by the aircraft, real-time requirements are met.

[0099] like Figure 5 As shown, the loss optimization module ultimately optimizes the dynamic space grid through its weighted loss function. Among the different loss terms, the weight of key obstacles is larger, and its weight can be amplified by 5 times compared to other loss terms. The weight of common obstacles is only amplified by 3 times, and the weight of the background area is only 0.3 times, so that the obstacle avoidance guidance process prioritizes high-threat obstacles.

[0100] The radar constraints are dynamically adjusted within this module. In the initial guidance phase, when the distance to the target is relatively far, the weight of the radar constraint loss can be as low as 0.3, increasing to 0.5 or higher as the target approaches, to adapt to the radar's accuracy characteristics at different distances. Path consistency verification avoids decision jumps and ensures the smoothness of the avoidance path by checking the differences in occupancy probabilities within a 3×3×3 neighborhood.

[0101] Ultimately, the loss optimization module can output early warning levels based on the optimized grid, and combine the radar distance and early warning accuracy of obstacles to provide clear decision-making basis for the aircraft, forming a decision result that includes the aircraft's maneuver commands.

Claims

1. A three-dimensional obstacle avoidance perception method for aircraft based on radar-vision fusion, characterized in that, Includes the following steps: S1. During the flight of the aircraft to the target, acquire guidance radar data from the external radar system and path image data from the guidance camera on the aircraft, and perform time synchronization and alignment. S2. For the synchronized guidance radar data and path image data, a pre-built lightweight feature fusion network is used to extract and cross-correlate features, and the fusion results in a multimodal feature map. S3. Based on the multimodal feature map and the current flight status data of the aircraft, generate a dynamic spatial grid that includes predicted flight path and detected obstacles; S4. Optimize the accuracy of the dynamic space grid using a preset loss function. Based on the optimized dynamic space grid, output the decision results containing the aircraft's maneuver commands to complete the aircraft's three-dimensional obstacle avoidance perception. In step S3, the current flight status data of the aircraft includes the geodetic pose of its guidance camera and the current velocity of the aircraft. Based on the multimodal feature map and flight status data, a three-dimensional space is divided within the same or different preset distance ranges corresponding to the forward, backward, left and right and near-ground sides of the predicted flight path, with the current position of the aircraft as the origin, thereby generating a dynamic spatial grid. The dynamic spatial grid adopts an adaptive resolution adjustment strategy. For the region that is far from the predicted flight path within a first preset distance range, a dense coding method is applied. For the region that is far from the predicted flight path within a second preset distance range, a sparse sampling method is applied. The maximum value of the first preset distance range is less than the minimum value of the second preset distance range. During the flight of the aircraft to the target, the dynamic space grid is incrementally updated according to the output of each frame of the multimodal feature map. The newly appearing grid in the three-dimensional space of the predicted flight path corresponding to each frame output is added, and the area outside the three-dimensional space that has been flown through after the displacement of the aircraft is deleted. For regions that are a third preset distance from the predicted flight path, full-resolution modeling is applied; the set corresponding to the third preset distance is a subset of the set corresponding to the first preset distance, and the minimum value of the third preset distance is equal to the minimum value of the first preset distance; when archiving any data frame, for regions outside the third preset distance, an octree structure compression method is used for data storage.

2. The three-dimensional obstacle avoidance perception method for aircraft according to claim 1, characterized in that: In step S1, the relative angle between the guidance camera and the aircraft is acquired simultaneously to characterize the pointing deviation of the guidance camera during the aircraft's flight to the target; at the same time, the aircraft's attitude information is acquired through the aircraft's IMU system for attitude correction when the aircraft performs maneuvers. When performing time synchronization alignment, the timestamp of the guidance radar data is used as a reference, and the delay of the path image data and the aircraft attitude information is compensated by interpolation. Then, the geodetic coordinate system pose of the guidance camera is calculated. This geodetic coordinate system pose is used to assist in the generation of the dynamic space grid in step S3.

3. The three-dimensional obstacle avoidance perception method for aircraft according to claim 2, characterized in that: When calculating the geodetic pose of the guidance camera, the proportional guidance method is used to determine the predicted flight path of the aircraft. Combined with the relative angle between the guidance camera and the aircraft, and the aircraft's attitude information, a direct calculation is performed, as shown in the following formula: In the formula, To guide the camera's geodetic pose; The attitude rotation matrix for guiding the aircraft is obtained based on the aircraft's attitude information; The relative rotation matrix of the guidance camera is obtained based on the relative angle between the guidance camera and the aircraft; When performing time synchronization alignment, for data frames in the guidance radar data whose jump distance exceeds the preset threshold, their timestamps are still used as the reference, but low confidence markers are added, and their corresponding weights are reduced when feature fusion is performed in step S2.

4. The three-dimensional obstacle avoidance perception method for aircraft according to claim 1, characterized in that: In step S2, based on the synchronized guidance radar data and path image data, the distance information in the guidance radar data is correlated with the pixel information in the path image data, and used as the input of the lightweight feature fusion network. The lightweight feature fusion network adopts the ShuffleNetV2 network, which uses the distance information in the guidance radar data as depth prior data, strengthens the region in the path image data that matches the guidance radar data through the spatial attention mechanism, and suppresses the region in the path image data that is unrelated to the flight path.

5. The three-dimensional obstacle avoidance perception method for aircraft according to claim 4, characterized in that: The fusion calculation formula for the ShuffleNetV2 network is as follows: In the formula, The fused multimodal feature map contains visual pixel information and radar distance information; Visual feature maps extracted from path image data by the ShuffleNetV2 network; This is a radar feature map obtained by converting data from the guidance radar. This is the cross function corresponding to the spatial attention mechanism. It calculates the similarity between visual features and radar features, with radar features as the dominant feature, and dynamically allocates feature weights during fusion. The higher the confidence level of the radar feature map, the higher its feature weight. If the confidence level of the radar feature map is lower than a preset threshold, then visual features are used as the dominant feature. The fused multimodal feature map shows the obstacle features on the flight path of the aircraft to the target, thus representing the pixel corresponding to the obstacle in the path image data and the radar distance corresponding to that pixel.

6. The three-dimensional obstacle avoidance perception method for aircraft according to claim 1, characterized in that: In step S4, the preset loss function is a weighted loss function, and its function is as follows: In the formula, to Preset weights for each loss term; For reprojection error loss of critical obstacles; This is the typical loss for dynamic spatial grids; Background loss; For radar constraint loss; This is the path consistency loss; As the aircraft gradually flies towards the target, when the distance between the aircraft and the target is less than a preset distance threshold, radar constraint loss occurs. weight Increase the preset value based on the initial value, and adaptively decrease the preset value of the sum of the remaining weights; path consistency loss. This is achieved through neighborhood convolution of a preset size.

7. The three-dimensional obstacle avoidance perception method for aircraft according to claim 6, characterized in that: Based on the optimized dynamic spatial grid, a dynamic threshold decision-making method is adopted. When an obstacle feature appears in the predicted area of ​​the dynamic spatial grid and the IoU value corresponding to the obstacle feature reaches the preset warning threshold, a multi-level warning decision is triggered. The multi-level early warning decision classifies obstacle risk levels into high risk, medium risk, and low risk based on the radar distance corresponding to different spatial grids that reach the preset early warning threshold. For high-risk levels, output hard evasion maneuver commands for the aircraft that include lateral acceleration constraints; For medium-risk levels, the output includes aircraft attitude change commands that include pitch angle adjustments; For low-risk levels, only proportional guidance is used to update the flight path planning and the predicted flight path of the aircraft.

8. A three-dimensional obstacle avoidance perception system for implementing the three-dimensional obstacle avoidance perception method for aircraft according to any one of claims 1 to 7, characterized in that, The system includes the following functional modules: The data synchronization module is used to acquire guidance radar data from external radar systems and path image data from guidance cameras on the aircraft, and to perform time synchronization and alignment. The feature fusion module is used to extract and cross-correlate features from guidance radar data and path image data using a pre-built lightweight feature fusion network, and obtain a multimodal feature map after fusion. The mesh generation module is used to generate a dynamic spatial mesh containing predicted flight paths and detected obstacles based on multimodal feature maps and the current flight status data of the aircraft. The loss optimization module is used to optimize the accuracy of the dynamic space grid using a preset loss function. Based on the optimized dynamic space grid, it outputs decision results containing aircraft maneuver commands.

Images