Dual-mode coupling sensing unmanned aerial vehicle obstacle avoidance control method and system

Abstract

The invention belongs to the technical field of unmanned aerial vehicle autonomous navigation and artificial intelligence. According to the dual-mode coupling sensing unmanned aerial vehicle obstacle avoidance control method and system, a visual image and laser radar point cloud data are fused, a two-dimensional detection frame, a category label and confidence of an obstacle are obtained through target detection, point cloud is projected to an image plane, and a corresponding point cloud set is screened by using the detection frame; extracting the three-dimensional centroid and geometric dimension of the obstacle, calculating the position and distance of the obstacle relative to the unmanned aerial vehicle, and sorting according to the distance; dynamically constructing a three-dimensional self-adaptive risk ellipsoid for the obstacles which are ranked in the front in combination with category semantic risk priori, geometric dimensions, relative positions, detection confidence coefficients and current velocity vectors of the unmanned aerial vehicles so as to represent dangerous areas of different obstacles; and inputting the self state, the target vector and the risk ellipsoid information into the reinforcement learning decision network, and outputting resultant force and a triaxial angular velocity instruction, thereby realizing safe, intelligent and efficient autonomous obstacle avoidance control.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation and artificial intelligence technology for unmanned aerial vehicles (UAVs), specifically to a dual-modal coupled perception method and system for UAV obstacle avoidance control. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Significant progress has been made in unmanned aerial vehicle (UAV) autonomous obstacle avoidance technology in recent years, with the deep integration of multi-sensor fusion and artificial intelligence becoming a key path to improve system performance. Visual sensors can provide rich semantic information and texture details, while LiDAR excels at acquiring precise 3D geometry; the two are naturally complementary at the perception level. Meanwhile, reinforcement learning, with its powerful policy search and decision-making capabilities, is widely used in UAV control in complex dynamic environments. Current research trends focus on building high-fidelity, high-consistency simulation training environments and enhancing the robustness and generalization ability of policies by fusing multimodal perception data, aiming to achieve efficient and safe transfer from simulation to reality.

[0004] However, existing technologies have fundamental shortcomings in obstacle risk modeling. Traditional methods typically use fixed shapes or single distance thresholds to define the danger zone of obstacles, failing to effectively integrate multi-dimensional heterogeneous information such as the semantic category, geometric dimensions, relative motion state, and perception confidence of obstacles. This static and coarse-grained risk representation cannot provide decision-making systems with a refined and adaptive understanding of the environmental situation, making it difficult for UAVs to generate differentiated and proactive avoidance strategies for obstacles of different risk levels, severely restricting the safety and intelligence of obstacle avoidance systems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a dual-modal coupled sensing method and system for UAV obstacle avoidance control, which fuses image information and radar information to output fixed-dimensional information, thereby improving the robustness and generalization capability of UAV obstacle avoidance control.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a dual-modal coupled sensing-based obstacle avoidance control method for unmanned aerial vehicles (UAVs).

[0007] A dual-modal coupled sensing method for obstacle avoidance control of unmanned aerial vehicles (UAVs) includes the following process: Acquire image data, lidar point cloud data, and the drone's own status data collected by the drone. The drone's own status data includes its current position and current velocity vector. Target detection is performed on the image data to obtain two-dimensional detection boxes, category labels and detection confidence of obstacles. The LiDAR point cloud data is then projected onto the image plane of the image data, and the point cloud sets falling within each two-dimensional detection box are selected using the two-dimensional detection boxes. Based on each point cloud set, the three-dimensional centroid and geometric dimensions of the corresponding obstacle are extracted. The relative position and relative distance of the obstacle to the drone are calculated based on the three-dimensional centroid and the current position of the drone. The detected obstacles are then sorted according to the relative distance. For at least one obstacle that ranks high in the order, a semantic risk prior coefficient is assigned based on the obstacle's category label. Then, a corresponding three-dimensional adaptive risk ellipsoid is constructed by combining the obstacle's geometric dimensions, the UAV's current velocity vector, relative position, and detection confidence. Each risk ellipsoid is characterized by the ellipsoid center, covariance matrix, and ellipsoid boundary threshold. The UAV's own state data, the relative target vector of the navigation target point, and the constructed three-dimensional adaptive risk ellipsoids are input into the decision network trained by reinforcement learning. The decision network outputs the resultant force and three-axis angular velocity used to control the UAV.

[0008] In one implementation of the first aspect of the present invention, extracting the three-dimensional centroid and geometric dimensions of the corresponding obstacle includes: Calculate the arithmetic mean of the three-dimensional coordinates of all points in the point cloud set that fall within the same two-dimensional detection box, and use the arithmetic mean as the three-dimensional centroid of the obstacle; For a set of point clouds falling within the same two-dimensional detection frame, the difference between the maximum and minimum values ​​is calculated on the three spatial coordinate axes to obtain the length, width, and height of the obstacle.

[0009] In one implementation of the first aspect of the present invention, constructing a corresponding three-dimensional adaptive risk ellipsoid includes: The three-dimensional centroid of the obstacle is weighted and fused with the current velocity vector of the UAV to determine the offset center of the risk ellipsoid as the center of the ellipsoid; The main axis of the covariance matrix is ​​constructed by taking the relative position of the obstacle and the drone as the main axis and correcting it by combining the current velocity direction of the drone. The basic axis distance is determined based on the geometry of the obstacle, and the semantic risk prior coefficient, the current speed of the drone, and the detection confidence are fused to determine the half-axis length of the covariance matrix.

[0010] In one implementation of the first aspect of the present invention, the UAV's own state data, the relative target vector of the navigation target point, and the constructed three-dimensional adaptive risk ellipsoids are input into a decision network trained by reinforcement learning, including: The parameters of each 3D adaptive risk ellipsoid are encoded using a shared encoder to obtain a latent vector of uniform dimension; all latent vectors are then averaged and pooled to obtain a risk situation fusion vector of fixed dimension. The drone's own state data, the relative target vector of the navigation target point, and the risk situation fusion vector are spliced ​​together to form the input vector of the decision network.

[0011] In one implementation of the first aspect of the present invention, the reward function used in the training of the decision network includes a risk ellipsoid region penalty term, which is calculated as follows: Calculate the Mahalanobis distance from the current position of the UAV to any three-dimensional adaptive risk ellipsoid, and apply a negative penalty based on the difference between the Mahalanobis distance and the ellipsoid boundary threshold when the Mahalanobis distance is less than the corresponding ellipsoid boundary threshold.

[0012] In one implementation of the first aspect of the present invention, the training process of the decision network is carried out in a pre-constructed simulation environment that integrates a 3D Gaussian splash scene and a physical mesh model. The simulation environment integrating the 3D Gaussian splash scene and the physical mesh model includes: Using the GS-SLAM method, a three-dimensional Gaussian splash scene composed of multiple three-dimensional Gaussian distributions is constructed. From the three-dimensional Gaussian splash scene, geometric consistency is optimized and LiDAR point cloud data is densified through differentiable rendering, and then a triangular mesh model is extracted. The triangular mesh model is simplified and decomposed into convex components, and after configuring physical properties for the triangular mesh model, it is imported into a physical simulation engine to form a physical mesh model.

[0013] In one implementation of the first aspect of the present invention, only obstacles with a relative distance less than a preset threshold are retained and arranged in ascending order according to the relative distance from smallest to largest. The top N obstacles are selected to construct a three-dimensional adaptive risk ellipsoid, where N≥1. The ellipsoidal boundary threshold is dynamically adjusted based on the detection confidence level, including: When the detection confidence is greater than the first threshold, the first ellipsoidal boundary threshold is used; When the detection confidence is less than the second threshold, the third ellipsoidal boundary threshold is used; When the detection confidence is between the first threshold and the second threshold, the second ellipsoidal boundary threshold is used, wherein the first threshold is greater than the second threshold, the first ellipsoidal boundary threshold is less than the second ellipsoidal boundary threshold, and the second ellipsoidal boundary threshold is less than the third ellipsoidal boundary threshold.

[0014] Secondly, the present invention provides a dual-modal coupled sensing obstacle avoidance control system for unmanned aerial vehicles.

[0015] A dual-modal coupled sensing obstacle avoidance control system for unmanned aerial vehicles (UAVs) includes: The data acquisition unit is configured to acquire image data, lidar point cloud data, and the drone's own state data collected by the drone, wherein the drone's own state data includes the drone's current position and current velocity vector. The multimodal fusion unit is configured to: perform target detection on the image data to obtain two-dimensional detection boxes, category labels and detection confidence of obstacles, project the LiDAR point cloud onto the image plane of the image data, and use the two-dimensional detection boxes to filter out the set of point clouds that fall within each two-dimensional detection box; The 3D information extraction unit is configured to: extract the 3D centroid and geometric dimensions of the corresponding obstacle based on each point cloud set, calculate the relative position and relative distance of the obstacle relative to the drone based on the 3D centroid and the current position of the drone, and then sort the detected obstacles according to the relative distance; The risk ellipsoid construction unit is configured to: assign semantic risk prior coefficients to at least one obstacle that ranks first, based on the obstacle's category label, and construct a corresponding three-dimensional adaptive risk ellipsoid by combining the obstacle's geometric dimensions, the UAV's current velocity vector, relative position, and detection confidence. Each risk ellipsoid is characterized by the ellipsoid center, covariance matrix, and ellipsoid boundary threshold. The decision control unit is configured to input the UAV's own state data, the relative target vector of the navigation target point, and the constructed three-dimensional adaptive risk ellipsoids into a decision network trained by reinforcement learning, and output the resultant force and three-axis angular velocity of the UAV to control the UAV.

[0016] Thirdly, the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the dual-modal coupled sensing obstacle avoidance control method for unmanned aerial vehicles according to the first aspect of the present invention.

[0017] Fourthly, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the dual-modal coupled sensing obstacle avoidance control method for unmanned aerial vehicles according to the first aspect of the present invention.

[0018] Compared with the prior art, the beneficial effects of the present invention are: This invention effectively addresses the technical problem in existing technologies where static and coarse-grained obstacle risk representations lead to a lack of targeted and forward-looking obstacle avoidance strategies. This method deeply integrates image semantic information with LiDAR point cloud geometric information. First, it uses target detection results to guide point cloud filtering, accurately extracting the 3D centroid and geometric dimensions of obstacles. Then, it introduces semantic risk prior coefficients and dynamically constructs a 3D adaptive risk ellipsoid by combining obstacle geometric features, relative position, UAV current speed, and detection confidence. This ellipsoid can adjust its shape, orientation, and coverage in real time according to obstacle attributes and flight status, achieving refined and differentiated modeling of obstacle danger zones. Finally, environmental information containing such high-order risk representations, along with the UAV's own state, is input into a reinforcement learning decision network, enabling the UAV to generate resultant force and angular velocity control commands that balance safety and efficiency based on a more accurate and comprehensive situational understanding. This significantly improves the UAV's autonomous obstacle avoidance capability and flight intelligence in complex dynamic scenarios.

[0019] This invention further enhances the safety and generalization ability of obstacle avoidance strategies through training mechanisms and simulation environment design, effectively addressing the disconnect between risk modeling and strategy learning in existing technologies. During training, a risk ellipsoid region penalty term based on Mahalanobis distance is introduced, enabling the reinforcement learning agent to accurately perceive its proximity to dynamic obstacle danger zones. Upon intrusion into the risk ellipsoid, it receives negative feedback matching the risk level, thus guiding the strategy to proactively avoid high-risk areas and improving flight safety. Simultaneously, the decision network is trained in a high-fidelity simulation environment that integrates a 3D Gaussian splash scene and a physical mesh model. This environment utilizes GS-SLAM to construct a Gaussian splash representation rich in semantic and geometric details, and optimizes the point cloud quality through differentiable rendering to extract structurally complete triangular meshes. After simplification, convex decomposition, and physical property configuration, the mesh is imported into the simulation engine, preserving the visual and geometric complexity of the real scene while meeting the computational efficiency and stability requirements of physical simulation. This virtual-real fusion training paradigm significantly improves the policy's transferability from simulation to reality, enabling UAVs to reliably perform refined obstacle avoidance tasks in real, complex environments.

[0020] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1A flowchart illustrating a dual-modal coupled sensing obstacle avoidance control method for unmanned aerial vehicles (UAVs) provided as an exemplary embodiment of the present invention; Figure 2 A schematic diagram of the reinforcement training process of a decision network provided as an exemplary embodiment of the present invention; Figure 3 A schematic diagram of a simplified mesh processing flow provided for an exemplary embodiment of the present invention; Figure 4 A schematic diagram of the adaptive ellipsoid generation process provided in an exemplary embodiment of the present invention; Figure 5 A schematic diagram comparing an obstacle ellipsoid and a risk area ellipsoid, provided as an exemplary embodiment of the present invention; Figure 6 A schematic diagram of a dual-modal coupled sensing UAV obstacle avoidance control system provided as an exemplary embodiment of the present invention; Figure 7 A schematic diagram of a computer device provided for an exemplary embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] Currently, the field of 3D reconstruction has proposed a novel scene representation method based on 3D Gaussian Splatting (3DGS). This method achieves high-quality new perspective synthesis and real-time rendering by modeling the scene as a large number of learnable Gaussian ellipsoids, demonstrating significant advantages in both visual realism and rendering efficiency. However, existing 3DGS methods are mainly geared towards static scene reconstruction and rendering tasks, lacking support for physical interaction processes and have not yet been applied to the construction of reinforcement learning-driven interactive simulation environments. Meanwhile, mesh reconstruction techniques have also made progress. For example, methods such as SuGaR and MILo can extract high-precision mesh structures from continuous scene representations, effectively preserving edge and detail features while maintaining geometric topological consistency. Although mesh models have been widely used in virtual manufacturing, medical imaging, and 3D printing, in existing UAV simulation environments, meshes are typically only used as geometric carriers for static collision detection, and their geometric information has not been fully utilized for state representation and decision optimization in reinforcement learning processes. In summary, existing technologies generally suffer from the following shortcomings: first, it is difficult to balance visual realism and physical accuracy; second, there is a lack of unified representation and coordination mechanisms between visual and geometric information; and third, simulation environments struggle to support differentiable optimization processes for reinforcement learning.

[0026] In view of this, this implementation proposes a dual-modal coupled perception-based UAV obstacle avoidance control method. By building a simulation scene that is closer to the real environment, it effectively solves the problem of insufficient visual effects in traditional reinforcement learning training environments. It can also solve the shortcomings of 3D Gaussian environments that lack real physical characteristics, improve the system's ability to transfer from simulation environments to real environments, effectively fuse image information and radar information to output fixed-dimensional information, enhance the system's robustness and generalization ability, and achieve efficient reinforcement learning training.

[0027] Specifically, such as Figure 1 As shown, the process includes the following: S101: Use the GS-slam method to build a 3D Gaussian scene, which is used to construct the visual layer of the environment; S102: Extracting Mesh and SDF from a 3D Gaussian field; S103: Integration of Mesh physicalization and simulation engine; S104: Multimodal sensing information fusion and adaptive risk zone construction; S105: Obstacle avoidance control action generation and execution; S106: Reward Function and Training Process.

[0028] like Figure 2As shown, this is a basic training process diagram. On the left, the basic model network, consisting of a UAV model and a 3D Gaussian network, provides underlying modeling support for the system. The RGB images and point cloud data collected by the UAV are input into the obstacle recognition and ellipsoid generation module, which outputs a multi-colored risk ellipsoid representing the obstacle in 3D space. On the other hand, the point cloud data, after pose estimation, is sent to the vector fusion and splicing module along with the ellipsoid information to complete feature integration. The fused feature input is processed by a reinforcement learning dual-network structure composed of an Actor network and a critic network to generate the optimal obstacle avoidance control decision. Finally, the control command is output to the flight control system, which then drives the UAV model and the 3D Gaussian network in reverse to form a closed-loop control, enabling the UAV to autonomously avoid obstacles in complex environments. In the application scenario of this invention, a 3DGS scene and a static mesh model are first constructed; then, the UAV acquires RGB images, radar point clouds, and its own state at the current moment; target detection and point cloud association are then performed to extract the geometric and semantic information of obstacles; next, priority obstacles are selected and a risk ellipsoid is generated; finally, the UAV state, risk ellipsoid parameters, and target point information are input into the reinforcement learning control network to output the UAV's resultant force and three-axis angular velocity, thereby achieving obstacle avoidance flight.

[0029] In step S101 of this implementation, specifically, it includes: The entire 3D scene is built using a combination of 3D Gaussian splashing and SLAM, a process that requires the use of a camera. First, the scene is modeled as a set of three-dimensional Gaussian distributions, combined with opacity parameters and spherical harmonic functions. The entire three-dimensional Gaussian distribution combination is represented by the following formula: (1); (2); Wherein, each Gaussian distribution G i , by location 3D covariance matrix Opacity And the spherical harmonic function corresponding to each color channel ( It is defined by a total of 12 coefficients; Represents the overall 3D Gaussian splash scene; The total number of Gaussian distributed cells in the scene; Representing the The probability density of a Gaussian distribution at a spatial point x; Representing the The opacity coefficient of a Gaussian sphere; Represents the coordinates of any point in three-dimensional space; Representing the The mean of a Gaussian distribution; The inverse matrix represents the i-th Gaussian covariance matrix; This represents transposition.

[0030] This invention employs an adaptive 3D Gaussian extended mapping technique, ensuring the stability of the mapping by updating and optimizing the 3D Gaussian scene at each selected keyframe. The entire process begins with obtaining the first frame image. First, this invention uniformly samples half the pixels from the entire image with a resolution of A×B and backprojects it into a 3D point X containing the corresponding depth observation value D. The method for constructing the 3D Gaussian scene representation is as follows: The position is set to X, and RGB color C is used... i Initialize the zero-degree spherical harmonic coefficients. Set the opacity parameter to a predefined value, and configure the covariance matrix based on the spatial point density. When subsequent images are received, this invention selects keyframes based on two criteria: one is the proportion of reliable regions in the currently observed image; the other is that if the relative pose of the current frame is too large compared to the previous keyframe, it will also be selected as a keyframe.

[0031] At each keyframe, this invention renders an RGB-D image of the current pose using a historical 3D Gaussian model, and then calculates the cumulative opacity of each pixel. If the cumulative opacity T of a pixel is too low or its rendering depth is too low, the calculation is performed accordingly. With observation depth If the difference is large, mark it as an unreliable pixel. These selected unreliable pixels primarily capture new observation areas. The present invention then backprojects these unreliable pixels into three-dimensional points. A new set of 3D Gaussian models is added to the scene representation to model the new observation region.

[0032] (3); (4); in, It is the first The opacity of a Gaussian sphere; It is the first The opacity of a Gaussian sphere; and These are the opacity threshold and the depth difference threshold, respectively. Represents the cumulative opacity per pixel; N represents the total number of Gaussian spheres. Represents the observed depth value; This represents the rendering depth value.

[0033] The invention then eliminates some outlier floating Gaussian spheres. After adding a new 3D Gaussian distribution, the invention checks all visible 3D Gaussian distributions in the current camera's view frustum and significantly reduces the opacity of 3D Gaussian distributions whose positions are not near the scene surface. Formally, for every visible three-dimensional Gaussian distribution, this invention can obtain its X. i Subsequently, the present invention can find the coordinates as The pixel at which the ray intersects the image plane and corresponds to the depth observation value D is determined using the following formula: (5); in, Represents the original Gaussian distribution; Represents the original opacity; Represents the corrected Gaussian distribution; Represents the opacity attenuation factor; Represents the position of the Gauss ball; Represents image pixels Corresponding three-dimensional points; This represents the threshold for determining depth difference.

[0034] When the difference between the depth of the Gaussian sphere and the depth observation value D is too large, this invention softly eliminates the Gaussian sphere by significantly reducing the opacity at that location. After completing the entire Gaussian map construction, this invention also performs necessary Bundle Adjustment (BA) optimization, uniformly selecting a portion of keyframes and simultaneously optimizing the global Gaussian parameters and camera pose, ultimately obtaining a 3DGS scene with high similarity to the real scene.

[0035] In step S102 of this implementation, specifically, it includes: After obtaining the entire 3D Gaussian scene, this invention uses the MVG-Splatting method for mesh extraction. To obtain accurate geometric depth, MVG-Splatting introduces geometric consistency constraints in differentiable rendering. For each pixel, the rendering depth value is obtained through volume accumulation. Ideally, the normal vector derived from the depth gradient should match the rendering normal vector N. (6); in, Indicates normalization; The partial derivative of the depth observation value D along the x-direction; The partial derivative of the depth observation value D along the y-direction is represented.

[0036] Mutual optimization loss Defined as: (7); in, and Represents the loss weighting coefficient; The gradient of the rendered image, compared with the gradient of the real image. To maintain consistency, depth is corrected using image edge information. By jointly optimizing Gaussian point properties, depth and normals mutually reinforce each other, gradually converging to the real surface. The sparse point cloud obtained in this process requires further compaction. This method uses photometric consistency to dynamically densen the point cloud, converting it into a scalar field on a regular voxel mesh. Linear interpolation is used to calculate the intersection points of isosurfaces, allowing the extraction of a high-precision triangular mesh. Thus, MVG-Splatting, through geometry optimization and compaction, enables classic Marching Cubes to generate richly detailed meshes from 3DGS.

[0037] This invention discretizes a continuous space into a regular mesh, and then uses interpolation to achieve continuous querying of any point, generating the SDF required by this invention. First, the entire mesh space is voxelized. For each voxel vertex, the minimum Euclidean distance to the mesh surface is calculated: (8); in, 't' represents a vertex of each voxel, and 't' represents any point on the Mesh. Usually, all triangles are traversed to find the shortest distance. This represents the minimum distance from a voxel vertex to the mesh surface. This represents a triangular mesh model. Next, the sign of the distance to points is determined using ray casting. Finally, for the distance value of each point in space, this invention requires trilinear interpolation of the eight vertices of its corresponding voxel, as shown in the following formula: (9); in, For trilinear interpolation weights, This represents the distance between the voxel vertex symbols. Let be the SDF value of spatial point p. This distance field is subsequently used for safety rewards, collision detection, near-obstacle penalties, and static risk constraints.

[0038] In step S103 of this implementation, specifically, it includes: By simplifying and decomposing the 3D mesh model and combining it with physical property configuration, the complex scene model is transformed from having only geometric representation capabilities into a physical entity with computable collision response and dynamic characteristics, thereby providing a stable and efficient physical interaction foundation for reinforcement learning training.

[0039] like Figure 3As shown, the simplification process includes: initializing the error matrix, calculating the edge folding cost, determining the optimal vertices and sorting them; then performing the operation of folding the first n edges, during which degenerate triangles and edges need to be deleted, and the cost and vertex position of the affected edges are updated; after completing the above operations, it is determined whether the target number of triangles has been reached. If not, the process returns to the step of folding the first n edges to continue iterating. If the target number has been reached, the process ends, thereby achieving efficient simplification of the 3D model.

[0040] First, this invention uses a Quadratic Error Metric (QEM) edge folding algorithm for simplification, calculating a 4×4 symmetric error matrix for each vertex in the Mesh. This matrix encodes the sum of squared distances from that vertex to all its adjacent triangular planes. For each edge, this invention calculates the cost Δa when folding the edge to a new vertex 'a', defined as the sum of squared distances from 'a' to its original adjacent planes. The formula is: (10); Among them, pp T It is a 4×4 matrix; Represents the cost of vertex folding; Represents the vertices to be collapsed; Represents the plane parameters of adjacent triangles; It represents the set of adjacent triangular planes.

[0041] This invention defines the sum of this matrix of all adjacent planes as the quadratic error matrix of that vertex. Then, by solving a system of linear equations, the optimal vertex position of each edge is obtained. All edges are sorted in ascending order of folding cost, and the edges with the lowest folding cost are folded first. Then, remove the degenerate triangles and edges, update the cost of the affected edges and the optimal vertex position, update the queue, and repeat the above steps. The specific process is as follows: Many physics engines and collision detection algorithms are inefficient or unable to handle concave objects directly. The goal of convex hull decomposition is to automatically divide a complex concave mesh into a set of non-intersecting convex polyhedra that approximate the original shape. This accurately represents complex shapes while leveraging efficient convex body collision detection algorithms.

[0042] The final mesh version was imported into the Gazebo platform to build a training scene, and its inertial properties were defined using an automatic calculation method. Collision properties and surface contact properties between objects were defined to give the simulation environment realistic collision and friction effects.

[0043] Next, the completed drone model needs to be imported into the simulation environment and equipped with an RGB camera and a Mid-360 LiDAR for perception.

[0044] In step S104 of this implementation, specifically, it includes: After obtaining the RGB image and point cloud information, an RGB image detection network is used to identify all obstacles within the current field of view, obtaining the 2D bounding box, category label, and detection confidence score for each obstacle. The current radar point cloud is projected onto the image plane, and the set of point clouds falling within the bounding boxes of each obstacle is counted. For each obstacle's corresponding point cloud set, the 3D centroid, length, width, height, point count, and relative distance to the UAV are extracted. The obstacles are sorted from closest to furthest from the UAV, and the n closest obstacles are selected. Based on the semantic category prior, geometric dimensions, UAV speed, and obstacle distance, the ellipsoidal semi-axis parameters of these n obstacles are calculated respectively. Using the UAV's current velocity direction as the principal axis direction of the ellipsoid, the covariance matrix corresponding to each obstacle is constructed, and the corresponding 3D adaptive risk ellipsoid is generated.

[0045] S104-1: RGB Image Target Detection and Semantic Recognition.

[0046] For the current image Perform target detection: (11); in, represent Set of target detection results for images at different times; Represents an object detection network; This represents the RGB image at time t.

[0047] No. Individual test results for: (12); For the first A two-dimensional detection box for an obstacle; For semantic category labels; To test the confidence level; This indicates the number of obstacles detected in the current image. After identification, the location and semantic information of each obstacle in the image are obtained.

[0048] S104-2: Point cloud projection onto the image plane.

[0049] The point cloud obtained by the lidar at that moment needs to be transformed into the camera coordinate system to facilitate point cloud classification and statistics according to obstacles.

[0050] Set point cloud set for: (13); in, Representing the A three-dimensional point cloud; Represents the total number of point clouds at time t; It represents the three-dimensional real number space.

[0051] After transforming each point from the point cloud coordinate system to the camera coordinate system, it is projected onto the image plane.

[0052] The point in the camera coordinate system is: (14); in, This represents the coordinates of the point cloud in the camera coordinate system; Represents the three-dimensional coordinate components in the camera coordinate system.

[0053] Based on the camera intrinsic parameter matrix The homogeneous projection can be obtained as follows: (15); After normalization, each 3D point is obtained. Corresponding to image pixel coordinates ; This represents the point cloud coordinates in the camera coordinate system.

[0054] S104-3: Calculate the point cloud within each obstacle box.

[0055] For the Within each detection bounding box, retrieve the point cloud containing all points falling within that bounding box. (16); Among them, the A collection of point clouds within a detection frame; Representative point cloud; Represents the pixel coordinates of the point cloud projection; Representing the A two-dimensional detection frame; represent Moment Cloud Collection.

[0056] To reduce background interference, depth consistency or clustering filtering can also be added.

[0057] S104-4: Extract the three-dimensional geometric properties of each obstacle.

[0058] Effective point cloud for each obstacle This invention extracts its three-dimensional dimensions, three-dimensional centroid, number of point clouds, and distance to the UAV.

[0059] For the three-dimensional centroid: (17); in: (18); This indicates the center position of all point clouds of the obstacle in three-dimensional space; This represents the number of point clouds within the i-th bounding box; Represents the point cloud within the frame; Represents the three-dimensional coordinate components of the centroid.

[0060] For three-dimensional dimensions, the length, width, and height of the obstacle are obtained by taking the range of the three-dimensional point cloud along the three coordinate axes: (19); (20); (twenty one); in, Represents the length of the obstacle; Represents the x-coordinate of the point cloud; Represents the y-coordinate of the point cloud; Represents the z-coordinate of the point cloud; Represents the width of the obstacle; Represents the height of the obstacle.

[0061] Let the obstacle size vector be: (twenty two); The distance from the obstacle to the drone is the same as the distance from the centroid of the point cloud to the drone.

[0062] S104-5: Select the nearest distance based on the distance. An obstacle.

[0063] All detected obstacles are sorted in ascending order of distance. This invention selects only the n obstacles closest to the target obstacle for ellipsoid generation. The value of n can be determined based on the density of obstacles in the scene. When there are more than five obstacles, this invention uses n=5; when there are fewer than five obstacles, this invention uses n=5. , This represents the current number of all obstacles.

[0064] S104-6: Assign semantic risk priors to each obstacle.

[0065] In the current environment, this invention uses semantic risk priors to assess the risk level of each obstacle. This invention defines a prior function. for: (twenty three); in, This represents the category of obstacle.

[0066] For each type of obstacle, this invention provides its risk coefficient: 0.9 for pedestrians, 0.85 for vehicles, 0.6 for boxes, 0.2 for walls, etc. This invention categorizes all prior function values ​​into five classes: (twenty four); in, Representing high mobility and high risk, 0.90–1.00; Representing the high-risk, mobile category, 0.75–0.89; This represents a static medium-risk class, with a risk level of 0.50–0.74. This represents a fixed low-to-medium risk level, ranging from 0.25 to 0.49. Representing the background / non-obstacle class, 0.00–0.24, and then specific categories are mapped to each level; This represents a set of risk levels.

[0067] S104-7: Construction of an adaptive ellipsoid.

[0068] like Figure 4 As shown, the input consists of two paths: a point cloud and an RGB image. The point cloud is first transformed to unify the spatial reference, while the RGB image outputs the location and semantic information of obstacles through object detection. The location information is used to statistically analyze the obstacle bounding box point cloud, and then the geometric attributes of the obstacles are extracted from it, which is then used to select... The system identifies the nearest obstacle and combines this information with the semantic information from the target detection output of the RGB image to generate an adaptive ellipsoid. This completes the three-dimensional spatial representation of the obstacle, providing accurate environmental perception input for subsequent UAV obstacle avoidance decisions.

[0069] For each selected obstacle This invention constructs a three-dimensional ellipsoid: (25); (26); in: Represents the center of the ellipsoid; Represents the covariance matrix; Represents the ellipsoid boundary threshold; Represents the k-th three-dimensional adaptive risk ellipsoid; Represents a point in space; Represents the initial center of the k-th ellipsoid; This represents the three-dimensional centroid of the k-th obstacle.

[0070] Let the velocity vector of the UAV be: (27); in, represent The drone's velocity vector at any given moment; Represents the three-axis velocity components.

[0071] Define the unit vector of velocity direction: (28); in, A unit vector representing the direction of velocity; This represents the minimum value.

[0072] The relative positions of the obstacle and the drone: (29); (30); (31); (32); in, This represents the position of the obstacle relative to the drone; Represents the geometric centroid of the obstacle; This represents the current location of the drone; Represents a unit vector of velocity; This represents the relative distance between the obstacle and the drone; It represents the cosine of the angle between velocity and relative position.

[0073] At this point, the invention updates the ellipsoid center to the control risk offset center, integrating position and velocity information: (33); in, Represents the risk ellipsoid offset center; and This represents the weighting coefficient.

[0074] Regarding the definition of the principal axis of the ellipsoid, this invention adopts the principle of relative position direction as the primary factor and velocity direction as the correction factor to construct a fused principal axis. The definition of the principal axis is as follows: (34); in, This represents the unit vector along the principal axis of the ellipsoid. This represents the direction correction factor.

[0075] Then construct the orthogonal product: , Representing the The unit vector in the direction of the first principal axis (major axis) of the risk ellipsoid; Representing the The unit vector in the direction of the second principal axis (central axis) of the risk ellipsoid; Representing the The unit vector in the direction of the third principal axis (minor axis) of a risk ellipsoid; Representing the The orthogonal rotation matrix corresponding to each obstacle.

[0076] like Figure 5 The diagram shows the obstacle risk ellipse fusion in the XY plane coordinate system, illustrating the optimization process of obstacle risk representation after fusing the drone's velocity direction. The black solid line in the diagram represents the original principal axis, corresponding to the blue original ellipse, which is the initial risk area generated solely based on the obstacle's geometry. When the drone moves at a speed V along the positive Y-axis, the velocity direction information is incorporated, and the principal axis is adjusted to the black dashed line (the principal axis after fusing the velocity direction). The corresponding risk area is updated to the red fused ellipse, achieving a more accurate and realistic dynamic representation of obstacle collision risk in the drone's motion state, providing more reasonable environmental constraints for drone obstacle avoidance decisions.

[0077] For the length of each half-axis, this invention incorporates semantic priors, velocity approximation, and confidence levels: (35); (36); (37); in, Represents the length of the x-axis of the ellipsoid; Represents the basic wheelbase of the obstacle; , and Represents the weighting coefficient; This represents prior semantic risk; Represents relative distance; Represents the length of the y-axis of the ellipsoid; Represents the base wheelbase; , and Represents the weighting coefficient; , and Represents the weighting coefficient; Represents the length of the z-axis of the ellipsoid; This represents the base wheelbase.

[0078] For the basic axis distance of the ellipse, the present invention obtains the minimum axis distances a, b, and c of the ellipse that can wrap around the obstacle through the minimum area outer ellipse method.

[0079] In step S105 of this implementation, specifically, it includes: For the decision layer of reinforcement learning, this invention uses an Actor-critic network for training. First, this invention concatenates the UAV's own pose vector with the target point and ellipsoid vector as the network input. The UAV's state is defined as follows: (38); in, , representing the location of the drone; , representing the linear velocity of the drone; , represents a quaternion; , representing angular velocity; This represents the state vector of the UAV at time t.

[0080] Navigation target point relative target vector : (39); The original risk region information contains many dimensions, such as location information, shape information, orientation information, semantic information, and confidence level information, forming an ellipsoidal risk feature region vector. This invention uses a shared encoder to encode each ellipsoidal parameter, obtaining a latent vector with uniform dimensions. : (40); in, These are the ellipsoidal parameters.

[0081] Then select The obstacles are averaged and pooled to balance the contribution of each obstacle to the current risk situation, resulting in a fixed-dimensional vector. : (41); in, Represents the first to nth ellipsoidal implicit vectors; This represents average pooling.

[0082] Finally, this invention yields the concatenated vector: (42) in, This represents the final input vector of the decision network.

[0083] Next, the concatenated vector is input into the network. After receiving the vector, the Actor outputs the continuous action distribution parameters, with the mean and variance as follows: (43); (44); in, Represents the mean of the action distribution; Represents the Actor policy network; Representative policy network parameters; Represents a variance prediction network; This represents the logarithm and variance of the action distribution.

[0084] The output four-dimensional action vector is as follows: (45); in, Represents the control action vector of the drone; Represents the angular velocity of the three axes; This represents the total combined force of the drones.

[0085] The output vectors are the total force and the three-axis angular velocities, respectively. Critic evaluates the value of the current state, outputting a scalar representing an estimate of the cumulative future reward from the current state.

[0086] The invention then transmits the obtained action output to the flight controller, causing the drone to move to the next moment and then enter the next round of perception-decision-modeling process.

[0087] To achieve safer and faster arrival at the target location, this invention provides a reward for the drone's actions. Reward Function The specific settings are as follows: rewards based on goal advancement. Collision penalty Risk ellipsoid penalty and motion smoothing rewards It consists of four parts: (46); Target progress reward To encourage drones to continuously approach the target point, it uses the form of "the difference in distance to the target between two consecutive moments": (47); in, Represents the location of the target point; This represents the drone's position at the previous moment; This represents the drone's position at this moment; The weight of the reward for achieving the target is represented.

[0088] Collision Penalty Used to strengthen flight safety constraints: (48); in, Represents the collision penalty value; This indicates a collision; This indicates that no collision occurred.

[0089] When the drone collides with the static grid environment, a large negative reward is given. This collision detection can be determined by the SDF field generated earlier in this invention. During training, it will push the policy to stay away from dangerous actions that lead to termination.

[0090] Risk ellipsoid region penalty: (49); in, The Mahalanobis distance from the drone to the risk ellipsoid; This is the threshold for the risk zone; This is the risk penalty weight.

[0091] Smooth motion penalty: (50); in, To smooth out the penalty coefficient; for Momentary action; for Momentary action.

[0092] Without a smoothing term, reinforcement learning can easily learn "jerky obstacle avoidance" or "sudden sharp turns," which may be possible in simulations but are not conducive to real flight control execution.

[0093] Below is the formula for the total reward: (51); For the overall training process, the PPO algorithm is used to update the parameters of the policy network and the value network. The parameter update process involves first collecting multiple trajectory segments and calculating the reward and advantage for each trajectory. The value network updates its parameters by minimizing the mean squared error loss through gradient descent, while the policy network updates its parameters by minimizing its negative value. The specific objective function is as follows: (52); (53); (54); (55); in, These are the parameters of the Critic network; It is Critic's prediction of the value of a state; It is the target reward obtained by accumulating trajectory rewards; The ratio of the probabilities of the new and old strategies; It is a computational advantage; These are the parameters of the Actor network; N represents the Critic network loss; N represents the number of samples. Represents state value prediction; Represents the cumulative return of the target; Represents the learning rate; Representative to gradient; Represents the loss in the Actor network; Represents the expected value of a time step; This represents the probability ratio between the old and new strategies; Represents the truncation function; Represents the cutoff coefficient; This represents the learning rate.

[0094] Ultimately, through extensive training in simulation environments, policy network and value network parameters that can be transferred to real-world environments at zero cost are obtained, thereby outputting safer and more stable navigation paths in real flight scenarios.

[0095] In summary, this invention achieves an organic combination of high-fidelity visual representation and precise physical constraints by constructing a unified simulation environment based on 3DGS, Mesh, and SDF, overcoming the problem of balancing visual realism and physical accuracy in existing reinforcement learning training environments. Simultaneously, by fusing RGB images, laser point clouds, semantic categories, geometric dimensions, and motion information, a three-dimensional adaptive risk ellipsoid is constructed and jointly designed with a reinforcement learning decision network and reward function, thereby improving the UAV's perception and avoidance capabilities of static obstacles and dynamic risks in complex scenarios. Furthermore, this invention effectively narrows the gap between the simulation environment and the real environment, improves the simulation-real generalization ability of the training strategy, reduces the number of real-machine experiments, lowers training costs, and enhances the safety, stability, and engineering feasibility of UAV autonomous navigation and control.

[0096] Figure 6 A dual-modal coupled sensing obstacle avoidance control system for unmanned aerial vehicles (UAVs) is shown, comprising: The data acquisition unit 601 is configured to acquire image data, lidar point cloud data and the drone's own status data collected by the drone, wherein the drone's own status data includes the drone's current position and current velocity vector. The multimodal fusion unit 602 is configured to: perform target detection on the image data to obtain two-dimensional detection boxes, category labels and detection confidence of obstacles, project the lidar point cloud data onto the image plane of the image data, and use the two-dimensional detection boxes to filter out the point cloud sets that fall within each two-dimensional detection box; The 3D information extraction unit 603 is configured to: extract the 3D centroid and geometric dimensions of the corresponding obstacle based on each point cloud set, calculate the relative position and relative distance of the obstacle to the drone based on the 3D centroid and the current position of the drone, and then sort the detected obstacles according to the relative distance. Risk ellipsoid construction unit 604 is configured to: assign semantic risk prior coefficients to at least one obstacle that ranks first, based on the obstacle's category label, and construct a corresponding three-dimensional adaptive risk ellipsoid by combining the obstacle's geometric dimensions, the UAV's current velocity vector, relative position, and detection confidence. Each risk ellipsoid is characterized by the ellipsoid center, covariance matrix, and ellipsoid boundary threshold. The decision control unit 605 is configured to input the UAV's own state data, the relative target vector of the navigation target point, and the constructed three-dimensional adaptive risk ellipsoids into a decision network trained by reinforcement learning, and output the resultant force and three-axis angular velocity of the UAV to control the UAV.

[0097] It is understood that the aforementioned units can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of the present invention. The aforementioned units are based on logical functional division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, the system may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0098] According to another embodiment of the present invention, the system of this embodiment can be constructed by running a computer program (including program code) capable of performing the steps involved in the corresponding method of the present invention on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, loaded into the aforementioned computing device through the computer-readable recording medium, and run therein.

[0099] Figure 7 A computer device is shown, which includes a processor 701, a communication interface 702, and a computer-readable storage medium 703. The processor 701, communication interface 702, and computer-readable storage medium 703 can be connected via a bus or other means.

[0100] The communication interface 702 is used to receive and send data. The computer-readable storage medium 703 can be stored in the memory of the electronic device. The computer-readable storage medium 703 is used to store computer programs, which include program instructions. The processor 701 is used to execute the program instructions stored in the computer-readable storage medium 703.

[0101] The processor 701 is the computing and control core of an electronic device. It is suitable for implementing one or more instructions, specifically for loading and executing one or more instructions to achieve the corresponding method flow or corresponding function.

[0102] Processor 701 is configured to perform the following procedure: Acquire image data, lidar point cloud data, and the drone's own status data collected by the drone. The drone's own status data includes its current position and current velocity vector. Target detection is performed on the image data to obtain two-dimensional detection boxes, category labels and detection confidence of obstacles. The LiDAR point cloud data is then projected onto the image plane of the image data, and the point cloud sets falling within each two-dimensional detection box are selected using the two-dimensional detection boxes. Based on each point cloud set, the three-dimensional centroid and geometric dimensions of the corresponding obstacle are extracted. The relative position and relative distance of the obstacle to the drone are calculated based on the three-dimensional centroid and the current position of the drone. The detected obstacles are then sorted according to the relative distance. For at least one obstacle that ranks high in the order, a semantic risk prior coefficient is assigned based on the obstacle's category label. Then, a corresponding three-dimensional adaptive risk ellipsoid is constructed by combining the obstacle's geometric dimensions, the UAV's current velocity vector, relative position, and detection confidence. Each risk ellipsoid is characterized by the ellipsoid center, covariance matrix, and ellipsoid boundary threshold. The UAV's own state data, the relative target vector of the navigation target point, and the constructed three-dimensional adaptive risk ellipsoids are input into the decision network trained by reinforcement learning. The decision network outputs the resultant force and three-axis angular velocity used to control the UAV.

[0103] This invention also provides a computer-readable storage medium, which is a memory device in an electronic device for storing programs and data. It is understood that the computer-readable storage medium here may include both built-in storage media in the electronic device and extended storage media supported by the electronic device. The computer-readable storage medium provides storage space for storing the processing system of the electronic device.

[0104] Furthermore, this storage space also contains one or more instructions suitable for loading and execution by the processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory; alternatively, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0105] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to perform the following process: Acquire image data, lidar point cloud data, and the drone's own status data collected by the drone. The drone's own status data includes its current position and current velocity vector. Target detection is performed on the image data to obtain two-dimensional detection boxes, category labels and detection confidence of obstacles. The LiDAR point cloud data is then projected onto the image plane of the image data, and the point cloud sets falling within each two-dimensional detection box are selected using the two-dimensional detection boxes. Based on each point cloud set, the three-dimensional centroid and geometric dimensions of the corresponding obstacle are extracted. The relative position and relative distance of the obstacle to the drone are calculated based on the three-dimensional centroid and the current position of the drone. The detected obstacles are then sorted according to the relative distance. For at least one obstacle that ranks high in the order, a semantic risk prior coefficient is assigned based on the obstacle's category label. Then, a corresponding three-dimensional adaptive risk ellipsoid is constructed by combining the obstacle's geometric dimensions, the UAV's current velocity vector, relative position, and detection confidence. Each risk ellipsoid is characterized by the ellipsoid center, covariance matrix, and ellipsoid boundary threshold. The UAV's own state data, the relative target vector of the navigation target point, and the constructed three-dimensional adaptive risk ellipsoids are input into the decision network trained by reinforcement learning. The decision network outputs the resultant force and three-axis angular velocity used to control the UAV.

[0106] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0107] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, digital cable) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dual-modal coupled sensing-based obstacle avoidance control method for unmanned aerial vehicles (UAVs), characterized in that, Includes the following processes: Acquire image data, lidar point cloud data, and the drone's own status data collected by the drone. The drone's own status data includes its current position and current velocity vector. Target detection is performed on the image data to obtain two-dimensional detection boxes, category labels and detection confidence of obstacles. The LiDAR point cloud data is then projected onto the image plane of the image data, and the point cloud sets falling within each two-dimensional detection box are selected using the two-dimensional detection boxes. Based on each point cloud set, the three-dimensional centroid and geometric dimensions of the corresponding obstacle are extracted. The relative position and relative distance of the obstacle to the drone are calculated based on the three-dimensional centroid and the current position of the drone. The detected obstacles are then sorted according to the relative distance. For at least one obstacle that ranks high in the order, a semantic risk prior coefficient is assigned based on the obstacle's category label. Then, a corresponding three-dimensional adaptive risk ellipsoid is constructed by combining the obstacle's geometric dimensions, the UAV's current velocity vector, relative position, and detection confidence. Each risk ellipsoid is characterized by the ellipsoid center, covariance matrix, and ellipsoid boundary threshold. The UAV's own state data, the relative target vector of the navigation target point, and the constructed three-dimensional adaptive risk ellipsoids are input into the decision network trained by reinforcement learning. The decision network outputs the resultant force and three-axis angular velocity used to control the UAV.

2. The dual-modal coupled sensing UAV obstacle avoidance control method as described in claim 1, characterized in that, Extract the three-dimensional centroid and geometric dimensions of the corresponding obstacle, including: Calculate the arithmetic mean of the three-dimensional coordinates of all points in the point cloud set that fall within the same two-dimensional detection box, and use the arithmetic mean as the three-dimensional centroid of the obstacle; For a set of point clouds falling within the same two-dimensional detection frame, the difference between the maximum and minimum values ​​is calculated on the three spatial coordinate axes to obtain the length, width, and height of the obstacle.

3. The dual-modal coupled sensing UAV obstacle avoidance control method as described in claim 1, characterized in that, Construct the corresponding three-dimensional adaptive risk ellipsoid, including: The three-dimensional centroid of the obstacle is weighted and fused with the current velocity vector of the UAV to determine the offset center of the risk ellipsoid as the center of the ellipsoid; The main axis of the covariance matrix is ​​constructed by taking the relative position of the obstacle and the drone as the main axis and correcting it by combining the current velocity direction of the drone. The basic axis distance is determined based on the geometry of the obstacle, and the semantic risk prior coefficient, the current speed of the drone, and the detection confidence are fused to determine the half-axis length of the covariance matrix.

4. The dual-modal coupled sensing UAV obstacle avoidance control method as described in claim 1, characterized in that, The UAV's own state data, the relative target vector of the navigation target point, and the constructed three-dimensional adaptive risk ellipsoids are input into the decision network trained by reinforcement learning, including: The parameters of each 3D adaptive risk ellipsoid are encoded using a shared encoder to obtain a latent vector of uniform dimension; all latent vectors are then averaged and pooled to obtain a risk situation fusion vector of fixed dimension. The drone's own state data, the relative target vector of the navigation target point, and the risk situation fusion vector are spliced ​​together to form the input vector of the decision network.

5. The dual-modal coupled sensing UAV obstacle avoidance control method as described in claim 1, characterized in that, During the training of the decision network, the reward function used includes a risk ellipsoid region penalty term, which is calculated as follows: Calculate the Mahalanobis distance from the current position of the UAV to any three-dimensional adaptive risk ellipsoid, and apply a negative penalty based on the difference between the Mahalanobis distance and the ellipsoid boundary threshold when the Mahalanobis distance is less than the corresponding ellipsoid boundary threshold.

6. The dual-modal coupled sensing UAV obstacle avoidance control method as described in claim 1, characterized in that, The training process of the decision network is carried out in a pre-constructed simulation environment that integrates a 3D Gaussian splash scene and a physical mesh model. This simulation environment includes: Using the GS-SLAM method, a three-dimensional Gaussian splash scene composed of multiple three-dimensional Gaussian distributions is constructed. From the three-dimensional Gaussian splash scene, geometric consistency is optimized and LiDAR point cloud data is densified through differentiable rendering, and then a triangular mesh model is extracted. The triangular mesh model is simplified and decomposed into convex components, and after configuring physical properties for the triangular mesh model, it is imported into a physical simulation engine to form a physical mesh model.

7. The dual-modal coupled sensing UAV obstacle avoidance control method as described in claim 1, characterized in that, Only obstacles with a relative distance less than a preset threshold are retained, and they are sorted in ascending order of relative distance from smallest to largest. The top N obstacles are selected to construct a three-dimensional adaptive risk ellipsoid, where N≥1. The ellipsoidal boundary threshold is dynamically adjusted based on the detection confidence level, including: When the detection confidence is greater than the first threshold, the first ellipsoidal boundary threshold is used; When the detection confidence is less than the second threshold, the third ellipsoidal boundary threshold is used; When the detection confidence is between the first threshold and the second threshold, the second ellipsoidal boundary threshold is used, wherein the first threshold is greater than the second threshold, the first ellipsoidal boundary threshold is less than the second ellipsoidal boundary threshold, and the second ellipsoidal boundary threshold is less than the third ellipsoidal boundary threshold.

8. A dual-modal coupled sensing obstacle avoidance control system for unmanned aerial vehicles (UAVs), characterized in that, include: The data acquisition unit is configured to acquire image data, lidar point cloud data, and the drone's own state data collected by the drone, wherein the drone's own state data includes the drone's current position and current velocity vector. The multimodal fusion unit is configured to: perform target detection on the image data to obtain two-dimensional detection boxes, category labels and detection confidence of obstacles, project the LiDAR point cloud data onto the image plane of the image data, and use the two-dimensional detection boxes to filter out the point cloud sets that fall within each two-dimensional detection box; The 3D information extraction unit is configured to: extract the 3D centroid and geometric dimensions of the corresponding obstacle based on each point cloud set, calculate the relative position and relative distance of the obstacle relative to the drone based on the 3D centroid and the current position of the drone, and then sort the detected obstacles according to the relative distance; The risk ellipsoid construction unit is configured to: assign semantic risk prior coefficients to at least one obstacle that ranks first, based on the obstacle's category label, and construct a corresponding three-dimensional adaptive risk ellipsoid by combining the obstacle's geometric dimensions, the UAV's current velocity vector, relative position, and detection confidence. Each risk ellipsoid is characterized by the ellipsoid center, covariance matrix, and ellipsoid boundary threshold. The decision control unit is configured to input the UAV's own state data, the relative target vector of the navigation target point, and the constructed three-dimensional adaptive risk ellipsoids into a decision network trained by reinforcement learning, and output the resultant force and three-axis angular velocity of the UAV to control the UAV.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1 to 7.

10. A computer device, characterized in that, include: Processor and computer-readable storage media; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the dual-modal coupled sensing obstacle avoidance control method for unmanned aerial vehicles as described in any one of claims 1 to 7.

Images