Aircraft obstacle avoidance method and device, storage medium and electronic equipment

Abstract

The invention relates to the technical field of aircraft control, and particularly provides an aircraft obstacle avoidance method and device, a storage medium and electronic equipment, and the method can comprise the steps: obtaining a future movement track of an obstacle in a flight region of an aircraft; constructing a space-time risk potential field based on the future movement track and the flight state information of the aircraft at the current moment; wherein the flight state information comprises an aircraft position and an aircraft speed; the space-time risk potential field represents the risk degree of collision between the obstacle and the aircraft in the physical space; the space-time risk potential field comprises a safe flight space; predicting a flight reference trajectory of the aircraft through the flight state information; and correcting the flight reference trajectory by using the space-time risk potential field to obtain an obstacle avoidance trajectory of the aircraft. According to the embodiment of the invention, the obstacle avoidance capability and precision of the aircraft can be improved.

Description

Technical Field

[0001] This application relates to the field of aircraft control technology, and more specifically, to a method, apparatus, storage medium, and electronic device for aircraft obstacle avoidance. Background Technology

[0002] In recent years, aircraft (such as drones) have been increasingly used in urban inspection, logistics transportation, emergency rescue, and low-altitude mapping, significantly increasing the complexity and dynamism of the flight environment. To ensure the safe operation of drones in unknown or dynamic environments, obstacle detection and obstacle avoidance decision-making technologies have become one of the core research directions.

[0003] Currently, traditional obstacle avoidance and detection solutions for aircraft typically rely on lidar or structured light sensors to acquire three-dimensional spatial information of the flight area, enabling obstacle detection and flight path planning. This means that current technologies primarily generate obstacle avoidance actions in real-time based on the detected obstacle positions in the current frame, which carries the risk of collisions due to delayed response. Clearly, current technologies lack the ability to predict the future movement trends of obstacles, preventing the aircraft from anticipating obstacle avoidance strategies, and thus their accuracy needs improvement.

[0004] Therefore, how to provide a precise method for aircraft obstacle avoidance has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of some embodiments of this application is to provide a method, apparatus, storage medium and electronic device for obstacle avoidance of an aircraft. The technical solutions of the embodiments of this application can enable the aircraft to know the obstacle avoidance plan in advance, ensure the timeliness and effectiveness of obstacle avoidance, and improve the accuracy of obstacle avoidance.

[0006] In a first aspect, some embodiments of this application provide a method for obstacle avoidance by an aircraft, comprising: acquiring the future motion trajectory of an obstacle in the flight area of ​​the aircraft; constructing a spatiotemporal risk potential field based on the future motion trajectory and the flight state information of the aircraft at the current moment; wherein the flight state information includes the aircraft position and the aircraft velocity; the spatiotemporal risk potential field characterizes the degree of risk of collision between the obstacle and the aircraft in physical space; the spatiotemporal risk potential field includes a safe flight space; predicting the flight reference trajectory of the aircraft using the flight state information; and correcting the flight reference trajectory using the spatiotemporal risk potential field to obtain the obstacle avoidance trajectory of the aircraft.

[0007] Some embodiments of this application construct a spatiotemporal risk potential field by predicting the future trajectory of obstacles and combining it with the flight state information of the aircraft. This spatiotemporal risk potential field is then used to correct the predicted flight reference trajectory of the aircraft, resulting in an obstacle avoidance trajectory. These embodiments allow the aircraft to anticipate obstacle avoidance plans in advance, ensuring the timeliness and effectiveness of obstacle avoidance and improving its accuracy.

[0008] In some embodiments, obtaining the future trajectory of an obstacle in the flight area of ​​the aircraft includes: acquiring dense point cloud data corresponding to the original image collected in real time by the aircraft; wherein the original image includes a left image frame and a right image frame; converting the dense point cloud data into a semantically enhanced point cloud; wherein the semantically enhanced point cloud includes the point cloud coordinates, semantic labels, and confidence scores of the obstacle; the semantic labels characterize the category of the obstacle; and predicting the obstacle based on the semantically enhanced point cloud to determine the future trajectory.

[0009] Some embodiments of this application convert dense point cloud data of the original image of the aircraft into semantically enhanced point cloud, and predict the trajectory of obstacles based on the semantically enhanced point cloud to obtain the future motion trajectory, thereby achieving accurate prediction of the motion trajectory of obstacles and providing effective support for subsequent predictive decision-based obstacle avoidance.

[0010] In some embodiments, acquiring dense point cloud data corresponding to the original image collected in real time by the aircraft includes: performing geometric correction on the original image to obtain a corrected image pair; wherein the corrected image pair includes a left corrected image and a right corrected image; calculating the disparity value of each pixel in the corrected image pair using a disparity cost function; calculating a depth map based on the disparity value; and obtaining the dense point cloud data by restoring the coordinates of the corrected depth map after the depth map correction.

[0011] Some embodiments of this application calculate the disparity value of each pixel by performing disparity calculation on the corrected image pair after geometric correction of the original image; after calculating the depth map by the disparity value, it is corrected and restored to obtain dense point cloud data, thereby realizing the three-dimensional representation of the image and improving data support for subsequent analysis.

[0012] In some embodiments, converting the dense point cloud data into a semantically enhanced point cloud includes: obtaining a pixel category probability map of any corrected image; wherein the any corrected image is a left corrected image or a right corrected image; inputting the any corrected image into a target detection network to obtain a set of target detection boxes; using the corrected depth map to backproject the pixels in each detection box in the target detection box set to the three-dimensional space of the dense point cloud data to obtain a local point cloud; after aligning the point coordinates in the local point cloud with the pixel coordinates in the any corrected image, projecting the pixel category probability map onto the local point cloud to obtain the semantically enhanced point cloud.

[0013] Some embodiments of this application process any calibrated image and any image frame to obtain a pixel category probability map and a set of target detection boxes; then, the local point cloud of each detection box is obtained, and combined with the pixel category probability map, a semantically enhanced point cloud is obtained, thereby realizing the classification of obstacles.

[0014] In some embodiments, predicting obstacles based on the semantically enhanced point cloud and determining the future motion trajectory includes: clustering the obstacles based on the semantic labels in the semantically enhanced point cloud to obtain the obstacle set at the current time; determining the obstacle state sequence of the obstacles in the obstacle set; constructing a spatiotemporal graph corresponding to the obstacle state sequence; and inputting the spatiotemporal graph into a spatiotemporal graph convolutional neural network to obtain the future motion trajectory.

[0015] Some embodiments of this application perform cluster analysis on obstacles using semantically enhanced point clouds to obtain obstacle state sequences, thereby constructing a spatiotemporal graph; finally, the spatiotemporal graph convolutional neural network is combined to achieve accurate and efficient prediction of the future motion trajectory of the obstacles.

[0016] In some embodiments, constructing a spatiotemporal risk potential field based on the future trajectory and the flight status information of the aircraft at the current moment includes: calculating the relative position and relative velocity of the aircraft and any obstacle among all obstacles at at least one future moment using the future trajectory and the flight status information; calculating the collision factor between the aircraft and any obstacle based on the relative position and the relative velocity; and calculating the collision factors of all obstacles to construct the spatiotemporal risk potential field.

[0017] Some embodiments of this application construct a spatiotemporal risk potential field by calculating the collision factors between the aircraft and all obstacles, so as to quantify the risk of the aircraft's flight area and reduce the collision risk.

[0018] In some embodiments, the safe flight space is obtained by defining the region in the spatiotemporal risk potential field where the risk value is less than a preset risk threshold as the safe flight space.

[0019] Some embodiments of this application determine a safe flight space by analyzing the risk value and preset risk threshold of the spatiotemporal risk potential field, thereby ensuring the safe flight of the aircraft.

[0020] In some embodiments, predicting the flight reference trajectory of the aircraft using the flight state information includes: inputting the flight state information and the original planned route of the aircraft into an upper-level predictive control model to obtain the flight reference trajectory of the aircraft at at least one future moment; wherein the flight reference trajectory includes flight position and flight parameters; and the upper-level predictive control model includes optimization functions and flight constraints.

[0021] Some embodiments of this application use an upper-level predictive control model to predict the flight trajectory of the aircraft at future moments to obtain a flight reference trajectory, providing data support for achieving precise obstacle avoidance.

[0022] In some embodiments, the step of using the spatiotemporal risk potential field to correct the flight reference trajectory and obtain the obstacle avoidance trajectory of the aircraft includes: inputting the flight state information, the flight reference trajectory and the spatiotemporal risk potential field into a lower-level predictive control model to obtain the obstacle avoidance trajectory; wherein, the lower-level predictive control model includes multiple optimization objective functions and flight constraints.

[0023] Some embodiments of this application refine the flight reference trajectory through a lower-level predictive control model and a spatiotemporal risk potential field to obtain a high-precision obstacle avoidance trajectory, enabling the aircraft to accurately avoid obstacles and greatly reducing the risk of collision.

[0024] Secondly, some embodiments of this application provide an obstacle avoidance device for an aircraft, comprising: an acquisition module for acquiring the future motion trajectory of an obstacle in the flight area of ​​the aircraft; a construction module for constructing a spatiotemporal risk potential field based on the future motion trajectory and the flight state information of the aircraft at the current moment; wherein the flight state information includes the aircraft position and the aircraft velocity; the spatiotemporal risk potential field characterizes the degree of risk of collision between the obstacle and the aircraft in physical space; the spatiotemporal risk potential field includes a safe flight space; a prediction module for predicting the flight reference trajectory of the aircraft using the flight state information; and a correction module for correcting the flight reference trajectory using the spatiotemporal risk potential field to obtain the obstacle avoidance trajectory of the aircraft.

[0025] Thirdly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0026] Fourthly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method as described in any embodiment of the first aspect.

[0027] Fifthly, some embodiments of this application provide a computer program product, the computer program product including a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 Flowcharts of an obstacle avoidance method for an aircraft provided for some embodiments of this application; Figure 2 System diagrams of aircraft obstacle avoidance provided for some embodiments of this application; Figure 3 Block diagrams of aircraft obstacle avoidance devices provided for some embodiments of this application; Figure 4 A schematic diagram of an electronic device provided for some embodiments of this application. Detailed Implementation

[0030] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0031] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] In related technologies, UAV obstacle avoidance methods are mainly divided into two categories: one is based on point clouds or depth sensors, directly acquiring three-dimensional spatial information through lidar or structured light sensors to achieve obstacle detection and path planning; the other is based on visual perception, relying on monocular or binocular cameras to extract image features and using neural networks for obstacle recognition and distance estimation. However, both of these methods have significant limitations in practical applications: most existing binocular visual obstacle avoidance methods only remain in a "detection-response" framework, that is, generating instant obstacle avoidance actions based on the obstacle position in the current frame, lacking prediction of the future movement trend of obstacles, and thus unable to provide predictive obstacle avoidance solutions for aircraft.

[0033] In view of this, some embodiments of this application provide a method for obstacle avoidance of an aircraft. This method first predicts the future trajectory of obstacles in the aircraft's flight area, and constructs a spatiotemporal risk potential field based on this prediction and the aircraft's flight state information. Then, the spatiotemporal risk potential field is used to correct the aircraft's flight reference trajectory to obtain the aircraft's obstacle avoidance trajectory. Embodiments of this application can achieve predictive decision-based obstacle avoidance for aircraft, reducing the risk of collisions.

[0034] The following is in conjunction with the appendix Figure 1 The implementation process of aircraft obstacle avoidance provided by some embodiments of this application is illustrated by way of example. It should be noted that the aircraft may be a drone or other aerial flying equipment, and the implementation process of aircraft obstacle avoidance may be executed by the processor within the drone's own system or by the server connected to the drone's communication. The embodiments of this application do not specifically limit this process.

[0035] like Figure 1 As shown, some embodiments of this application provide a flowchart of a method for obstacle avoidance by an aircraft. This method for obstacle avoidance by an aircraft may include: S110, acquire the future trajectory of obstacles in the flight area of ​​the aircraft.

[0036] For example, in specific embodiments of this application, a pre-trained model can be used to directly predict the future trajectory of an obstacle, or the future trajectory can be predicted after analyzing the specific circumstances of the obstacle.

[0037] In some embodiments of this application, S110 may include: S111, acquire dense point cloud data corresponding to the original image collected in real time by the aircraft; wherein, the original image includes a left image frame and a right image frame.

[0038] For example, in a specific embodiment of this application, the onboard binocular camera of the UAV first simultaneously acquires two image frames, one on the left and one on the right. L (x,y) and I R(x,y). Then, for I... L (x,y) and I R The (x,y) data is processed and analyzed to obtain dense point cloud data.

[0039] In some embodiments of this application, S111 may include: S1111, Perform geometric correction on the original image to obtain a corrected image pair; wherein, the corrected image pair includes a left corrected image and a right corrected image.

[0040] For example, in a specific embodiment of this application, system calibration parameters are used to calibrate I. L (x,y) and I R Geometric correction is performed on (x,y). This is achieved using the correction matrix R. rect With projection matrix P L P R This ensures that corresponding pixels in the two images lie on the same horizontal scan line, thus simplifying the 3D depth estimation problem into a one-dimensional disparity problem. After the above correction process, the corrected image pair is output. (i.e., left-corrected image) and (i.e., right-corrected image).

[0041] S1112, calculate the disparity value of each pixel in the corrected image pair using the disparity cost function.

[0042] For example, in a specific embodiment of this application, an adaptive matching resolution mechanism is introduced. This mechanism determines the matching step size and window size for analyzing the midpoint pixels of the corrected image, ensuring pixel resolution. Specifically, the matching step size and window size are dynamically adjusted based on the UAV's current flight speed, reducing computational load during high-speed flight and improving accuracy during low-speed or hovering flight. The specific adjustment formula is as follows:

[0043] in, W The current window size. W 0 is the default window size. v ref For the reference speed of the drone, k v This is the adjustment coefficient.

[0044] disparity cost function E ( d An improved semi-global matching form is adopted, and its formula is as follows:

[0045] Where λ is the weight of the regularization term (a non-negative scalar), used to balance data consistency and disparity field spatial smoothness. E ( d Let be the cost function defined for the disparity map of the entire image. By minimizing the cost function, the pixel coordinates can be obtained as ( x , y disparity value d ( x , y ).

[0046] Understandably, the parallax cost function can be flexibly adjusted according to actual needs.

[0047] S1113, calculate the depth map by calculating the disparity value.

[0048] For example, in a specific embodiment of this application, the depth map is calculated by the ratio of the product of the focal length and baseline length of the binocular camera to the disparity value. The formula for calculating the depth map is as follows:

[0049] in, f Focal length B Baseline length; depth map D ( x , y The unit is meters.

[0050] S1114, The dense point cloud data is obtained by restoring the coordinates of the corrected depth map after the depth map correction.

[0051] For example, in a specific embodiment of this application, to improve the stability and noise resistance of the depth results, the system performs depth confidence screening and spatial reconstruction on the generated depth map. The depth confidence is based on the matching residual C(…) of the left and right images. x , y ) is determined; among which, the matching residual C( x , y The formula is as follows:

[0052] in, The brightness difference / residual scale parameter (which is a scalar) is used to convert pixel grayscale differences into confidence levels. It can control the sensitivity of the confidence function to grayscale differences and can be set empirically or statistically (such as the standard deviation of local brightness in an image).

[0053] When C( x , y (Below the set threshold) At this point, the depth map is corrected using the neighborhood mean correction method to obtain the corrected depth map. The formula for neighborhood mean correction is as follows:

[0054] Where N is the number of pixels.

[0055] Then, based on the corrected depth map D'( x , y The pixel coordinates are then converted back to 3D points in the camera coordinate system, resulting in dense point cloud data P, where P = {X, Y, Z}. The formula for this conversion is:

[0056] Among them, c x and c y These are the positions of the principal point of the camera (i.e., a stereo camera) in pixel coordinates. They are camera intrinsic parameters used to back-project the pixel coordinates onto the camera coordinate system.

[0057] S112, the dense point cloud data is converted into a semantically enhanced point cloud; wherein, the semantically enhanced point cloud includes the point cloud coordinates, semantic labels and confidence scores of the obstacles; the semantic labels represent the category of the obstacles.

[0058] For example, in a specific embodiment of this application, information with category semantics is extracted from the image and 3D point cloud obtained by binocular vision reconstruction. Specifically, the original image is converted into a 3D semantic point cloud with category labels (i.e., semantic labels) (as a specific example of a semantically enhanced point cloud), providing a high-level scene understanding for subsequent trajectory prediction.

[0059] In some embodiments of this application, S112 may include: S1121, Obtain the pixel category probability map of any corrected image; wherein, the any corrected image is a left corrected image or a right corrected image.

[0060] For example, in a specific embodiment of this application, any one of the corrected images is selected from the corrected image pair for semantic segmentation. For example, taking the left corrected image as an example, a semantic segmenter is used to extract feature maps from the left corrected image to obtain feature map F. s , .

[0061] in, For semantic segmentation encoders, classic semantic segmentation models (e.g., the backbone network of the CLIP model) can generally be used. For the extracted feature map, H 'and W 'These are the spatial dimensions (i.e., height and width) after downsampling.C This represents the number of semantic feature channels.

[0062] Next, a pixel-level classification network (e.g., implemented using convolution and softmax) is used to perform semantic segmentation output; the formula for the classification network is as follows:

[0063] in, S ( x , y , c ) is a pixel category probability map, which represents the pixel ( x , y ) belongs to category c The probability of W; c The weights represent the classification network. Represents the convolution operation; Softmax( () is the softmax function. A semantic label map is obtained by matching each pixel to the category with the highest probability. L s ( x , y ), used to represent the category value of each pixel.

[0064] S1122, Input any of the corrected images into the target detection network to obtain a set of target detection boxes.

[0065] For example, in a specific embodiment of this application, after obtaining the pixel category probability map, the system also needs to identify the position and outline of the obstacle target to further support dynamic obstacle prediction. First, the left-corrected image is input into the target detection network to obtain a set of target detection boxes. The formula for the target detection network is: ;in, For object detection models (such as YOLO v8 or CenterNet); The weights of a trainable object detection model; B ={ b i} represents the set of target detection boxes; where each detection box... b i =( x i , y i , w i , h i , c i , p i );in,( x i ,y i ) represents the coordinates of the center of the detection box; w i , h i () represents the width and height of the detection frame; c i For the target category, p i This represents the confidence level probability.

[0066] S1123, using the corrected depth map, the pixels in each detection box in the target detection box set are back-projected to the three-dimensional space of the dense point cloud data to obtain a local point cloud.

[0067] For example, in a specific embodiment of this application, a modified depth map is used to estimate the three-dimensional position of each target. D '( x, y The pixels within the detection bounding box are back-projected onto a 3D space P={X,Y,Z} to obtain a local point cloud. P i : P i ={ X , Y , Z |( x, y )∈ b i}

[0068] Then, the 3D bounding box is obtained by fitting the minimum bounding cube:

[0069] in,( x c , y c , z c () is the center of the bounding box; l , w , h ( ) represents the length, width, and height of the bounding box; The bounding box's orientation angle. Output a list of 3D targets { This will be used for subsequent point cloud alignment and fusion.

[0070] S1124, after aligning the point coordinates in the local point cloud with the pixel coordinates in any of the corrected images, the pixel category probability map is projected onto the local point cloud to obtain a semantically enhanced point cloud.

[0071] For example, in a specific embodiment of this application, a left-corrected image and a corrected depth map were obtained through the above embodiments. D '( x, y ), local point cloudP ={( X i , Y i , Z i ) }; and the camera's intrinsic and extrinsic parameters (K); R ext , T ext The internal parameter K is:

[0072] in, f x and f y Focal length c x and c y Main point, ( R ext , T ext ) represents the rotation and translation of the camera to the point cloud coordinate system.

[0073] Local point cloud data P ={( X i , Y i , Z i The input is fed into the point cloud encoder to obtain geometric feature vectors, thereby realizing the point cloud features in the local point cloud. F p Feature map of the left-corrected image F s Having the same dimensions facilitates subsequent alignment. Specifically, the point cloud encoder is:

[0074] in, For point cloud encoders, the PointNet++ model can be used; For encoder parameters, F p ( X i , Y i , Z i )∈R C Let be the geometric eigenvector of point i.

[0075] To ensure consistency between image semantic features and point cloud geometric features in feature space, a camera projection model is first used to project the point cloud within the 3D bounding box back onto the image plane, where, ( X i , Y i , Z i () represents the coordinates of the i-th point in the local point cloud. u i , v i Let be the pixel coordinates of the point in the left-corrected image, and its transformation alignment formula is:

[0076] The alignment described above yields two sets of feature pairs that correspond to each other in the same spatial location: ( F s ( u i , v i ), F p ( X i , Y i , Z i )).

[0077] Then, cross-modal consistency loss is used to bring the feature embeddings of the two modalities closer together:

[0078] Where ||·||2 is the Euclidean distance norm; N p The loss is the number of points involved in registration; this loss is used to simultaneously optimize the semantic segmentation encoder parameters in the image through gradient backpropagation. Point cloud encoder parameters This makes the feature space distribution of the two outputs more consistent.

[0079] After aligning image features with point cloud features using the above method, the semantic probabilities in the image can be... S ( u , v , c Projecting the image onto the local point cloud yields a semantically enhanced point cloud. P f ={ X i , Y i , Z i , , p i Each 3D point carries semantic tags in addition to its geometric coordinates. With confidence level p i This information will be used for subsequent motion state estimation.

[0080] S113, Based on the semantically enhanced point cloud, predict the obstacle and determine the future motion trajectory.

[0081] For example, in a specific embodiment of this application, based on semantically enhanced point clouds P f This paper estimates the future trajectories of dynamic obstacles in the environment. Unlike traditional prediction methods based on geometry or filtering, this application employs a spatiotemporal graphical neural network (ST-GNN) to simultaneously model temporal evolution and multi-object spatial interaction within a unified framework, thereby achieving high-precision prediction of the short-term trajectories of dynamic obstacles.

[0082] In some embodiments of this application, S113 may include: S1131, Cluster the obstacles based on the semantic labels in the semantically enhanced point cloud to obtain the obstacle set at the current moment.

[0083] For example, in a specific embodiment of this application, when obtaining a semantically enhanced point cloud... P f Next, we'll start with semantic categories (as a specific example of semantic tags). Clustering is performed to obtain the set of obstacles at time t (as a specific example of the current time). O t : O t ={ O 1, O 2, O 3,..., O M}; where M represents the number of obstacles detected in the current frame.

[0084] S1132, determine the obstacle state sequence of the obstacles in the obstacle set.

[0085] For example, in a specific embodiment of this application, for each obstacle O i Calculate the geometric center of its point cloud. Q i Using as its center point, we can then obtain the set of obstacle centers at time t. : ={ Q1(t), Q 2(t),..., Q M (t)}; Furthermore, the obstacle state sequence A for consecutive T frames can be obtained: A={ ,..., }

[0086] S1133, Construct a spatiotemporal graph corresponding to the state sequence of the obstacles.

[0087] For example, in a specific embodiment of this application, to characterize the temporal and spatial dependencies between obstacles, the aforementioned obstacle state sequence is constructed as a spatiotemporal graph G=(V, E). The set of nodes V is as follows:

[0088] Each node represents the state of obstacle target j at time t0.

[0089] Features of each node for:

[0090] Among them, OneHot ( ) represents the OneHot vector of semantic category.

[0091] Besides nodes, a spacetime graph also includes edges, which are further divided into temporal edges and spatial edges. Spatial edges For obstacles j and k:

[0092] in, d th This represents the threshold for adjacency distance; that is, at time t, the distance between obstacles j and k is less than the threshold. d th When, then the space boundary exist.

[0093] For time edge This represents the connection between adjacent time frames for the same target.

[0094] S1134, The spatiotemporal graph is input into the spatiotemporal graph convolutional neural network to obtain the future motion trajectory.

[0095] For example, in a specific embodiment of this application, a spatiotemporal graph convolutional neural network (ST-GNN) is used to perform feature propagation and trajectory prediction on the spatiotemporal graph G. This network consists of three parts: a graph convolutional layer, a temporal convolutional layer, and a fully connected layer. The graph convolutional layer is used to capture spatial interactions between obstacles; the temporal convolutional layer is used to capture historical motion trends; and the fully connected layer is used to generate the final predicted temporal data. If... If we represent a spatiotemporal graph convolutional neural network model, then the operation of trajectory prediction using it can be represented as follows:

[0096] Among them, M GC ( ) represents a graph convolutional layer; M TC ( ) represents a temporal convolutional layer; M FC ( ) represents a fully connected layer. That is, the spatiotemporal graph data G will be processed through 2 graph convolutional layers, 1 temporal convolutional layer, and 1 fully connected layer to obtain the final trajectory prediction result (as a specific example of future motion trajectory).

[0097] The following example illustrates the specific implementation process of these three layers.

[0098] For graph convolutional layers, they are used to process nodes. The spatial features are updated, and the update method is as follows:

[0099] in, Indicates the first l The input features of the layer Represents a node The set of all neighboring nodes; W s and W n The weight matrix is ​​the trainable weight matrix; ReLU( ) represents the ReLU activation function.

[0100] For the temporal convolutional layer, a combination of temporal windows and one-dimensional convolution Conv1D(·) is used:

[0101] in, This represents the features extracted by the temporal convolutional layer. This represents the length of the time window.

[0102] Generating time series of future trajectories using fully connected layers:

[0103] in, This represents the position of target j in the k-th frame; K represents the length of the prediction time domain. W 0 and b Trainable weights and biases for a fully connected layer.

[0104] Finally, by summing up the position information of all targets in the next k frames, we can obtain the future trajectory information (i.e., the future motion trajectory). :

[0105] S120, based on the future trajectory and the flight status information of the aircraft at the current moment, a spatiotemporal risk potential field is constructed; wherein, the flight status information includes the aircraft position and the aircraft speed; the spatiotemporal risk potential field characterizes the degree of risk of the obstacle colliding with the aircraft in physical space; the spatiotemporal risk potential field includes a safe flight space.

[0106] For example, in a specific embodiment of this application, the future motion trajectory of the obstacle is... This is transformed into a risk potential field distribution and dynamic feasible space constraints. Specifically, by calculating the future relative states of the aircraft and obstacles, a spatiotemporal risk potential field that dynamically changes over time is generated. Based on this, a safe flight domain (i.e., safe flight space) is extracted to provide constraint boundaries and cost terms for subsequent predictive control modules. Unlike traditional fixed-threshold obstacle avoidance methods, this application uses a spatiotemporal risk potential field modeling and probabilistic safe domain generation mechanism, which can explicitly characterize prediction uncertainty and temporal dynamics.

[0107] In some embodiments of this application, S120 may include: S121, using the future trajectory and the flight state information, calculate the relative position and relative speed between the aircraft and any obstacle among all obstacles at at least one future moment.

[0108] For example, in a specific embodiment of this application, the current flight status information of the drone is obtained in real time:

[0109] in, x e ( t ), y e ( t ), z e ( t ) represent the position of the UAV at time t (i.e., its three-dimensional coordinates); while v x (t ), v y ( t ), v z ( t ) represent velocities in three dimensions.

[0110] For each predicted time t+k (as a specific example of a future time), calculate the relative position of the drone with the j-th obstacle. p j and relative velocity v j :

[0111]

[0112]

[0113] in, This represents the predicted position of the drone at time t+k (obtained by short-time linear interpolation of the drone's position). This represents the predicted velocity of the drone at time t+k.

[0114] Let represent the predicted velocity of the obstacle at time t+k, which is obtained by approximating the future trajectory using the differential:

[0115] S122, calculate the collision factor between the aircraft and any obstacle based on the relative position and the relative velocity.

[0116] For example, in a specific embodiment of this application, the instantaneous collision factor when the UAV collides with obstacle j (i.e., any obstacle) at time t+k is obtained by the following formula. r j (t+k):

[0117] Where exp(·) represents the exponentiation operation, λ p and λ v Risk scale weights representing position and velocity.

[0118] S123, calculate the collision factors of all obstacles to construct the spatiotemporal risk potential field.

[0119] For example, in a specific embodiment of this application, a continuous spatiotemporal risk potential field is constructed by weighting and superimposing the collision factors of all obstacles.U ( x , y , z , t + k ):

[0120] in, α j The weight representing the importance of an obstacle reflects the radius and intensity of its risk impact in physical space (for example, it can be determined based on semantic categories, such as "large aircraft" having a greater weight than "small aircraft"). The higher the value in this spatiotemporal risk potential field, the higher the risk in that area, and the more it should be avoided.

[0121] In some embodiments of this application, the safe flight space is obtained by defining the region in the spatiotemporal risk potential field whose risk value is less than a preset risk threshold as the safe flight space.

[0122] For example, in a specific embodiment of this application, a preset risk threshold is set. θ safe The dynamic feasible space (as a specific example of a safe flight space) is then:

[0123] in, (t+k) represents the set of three-dimensional spaces (i.e., safe flight space) that the UAV can safely pass through at time t+k.

[0124] S130, predict the flight reference trajectory of the aircraft using the flight status information.

[0125] For example, in a specific embodiment of this application, before predicting the flight reference trajectory, a path function is generated based on the path modeled by discrete motion state.

[0126] Given the known flight status information of the drone X ego (t) and control input u ( t )=[ a x ( t ), a y ( t ), a z ( t In the case of [ ], a discrete linear dynamics model is used for prediction. Among them, the control behavior u ( tThis refers to the acceleration command applied to the UAV in the three axes at time t. The formula for the discrete linear dynamics model is as follows:

[0127] in, This represents matrix multiplication, where A and B are matrices.

[0128]

[0129] in, I is the time step, and I3 is a 3×3 identity matrix.

[0130] For a drone, its state prediction for the next k steps can be expressed as:

[0131] The method described above for predicting time from t+1 to t+K is called the path generation method. f dyn (·)

[0132] This path generation method can be used to generate flight reference trajectories for drones.

[0133] In some embodiments of this application, S130 may include: inputting the flight state information and the original planned route of the aircraft into the upper-level predictive control model to obtain the flight reference trajectory of the aircraft at at least one future moment; wherein the flight reference trajectory includes flight position and flight parameters; and the upper-level predictive control model includes optimization functions and flight constraints.

[0134] For example, in a specific embodiment of this application, a top-level predictive control model (Top MPC) is used to perform task-level path guidance over a longer time domain, generating a smooth reference trajectory for the UAV. (i.e., flight reference trajectory). The specific generation formula is:

[0135] in, f guide (·) represents the control strategy within the upper-level predictive control model.

[0136] Specifically, the current flight status information X ego (t) and the mission objective X goal As input to the upper-level predictive control model; where, X goalThis represents the initial planned flight path of the UAV (i.e., where the UAV should be located at what time). The output of the upper-level predictive control model is the reference trajectory. k=1,2,3,...,N c This reference trajectory represents the drone's trajectory in the future N, calculated based on rough constraints. c The flight position and flight parameters at a step (or future moment).

[0137] The optimization function of the upper-level predictive control model is:

[0138]

[0139] The flight constraints (the strength of the constraint control behavior and ensuring that the flight path remains within the passable area) are as follows:

[0140] in, U global Indicates control behavior. λ curve This represents the weight. The goal of this optimization is to make the flight path fit as closely as possible to the original planned route (the first constraint, i.e., ...). X ego and X goal To minimize the deviation and make the generated trajectory as smooth as possible (i.e., the second constraint), the goal is to achieve the optimization objective while obtaining... U global After that, U global The input is fed into the dynamic model of the upper-level predictive control model to obtain the global reference trajectory:

[0141] S140, the flight reference trajectory is corrected using the spatiotemporal risk potential field to obtain the obstacle avoidance trajectory of the aircraft.

[0142] For example, in a specific embodiment of this application, the lower-level predictive control model can dynamically correct the reference trajectory according to the following formula within a shorter time domain, in conjunction with the spatiotemporal risk potential field, to optimize the control input, wherein, f react ( This is for local optimization.

[0143] In some embodiments of this application, S140 may include: inputting the flight state information, the flight reference trajectory, and the spatiotemporal risk potential field into the lower-level predictive control model to obtain the obstacle avoidance trajectory; wherein, the lower-level predictive control model includes multiple optimization objective functions and flight constraints.

[0144] For example, in a specific embodiment of this application, the Bottom Predictive Control Model (Bottom MPC) is optimized in real time over a shorter time domain to achieve tracking of the upper-level trajectory and local obstacle avoidance.

[0145] Specifically, the input to Bottom MPC is flight status information. X ego (t), reference trajectory and the spacetime risk field U ( x , y , z , t + k Its output is a fine-grained control sequence that needs to be executed by the drone (as a specific example of an obstacle avoidance trajectory):

[0146] Specifically, the internal multinomial optimization objective function of Bottom MPC is as follows:

[0147] The flight constraints are:

[0148] As can be seen from the multiple optimization objective functions, the following will be achieved: X ego (t+k) and The comparison is made because the reference trajectory is compared to the originally planned route. X goal More precise; in addition, it incorporates risk loss prevention measures. J risk and energy loss J energy and their corresponding weights λ risk and λ energy Specifically, J energy The formula is:

[0149] J risk The formula is:

[0150] in, X ego (t+k) represents the short-term predicted position of the UAV at time t+k. β j For a certain type of obstacle, the "weighting term during decision optimization" (i.e., the drone's "sensitivity to risk": that is, whether the drone will immediately avoid the obstacle or get slightly closer is fine). U ( ) represents the risk value obtained from the spatiotemporal risk potential field.

[0151] It should be noted that in the actual process of generating UAV control behavior, a two-layer predictive control model (TopMPC and Bottom MPC) works together. Specifically, the upper-layer predictive control model generates the reference trajectory at larger time intervals, while the lower-layer predictive control model generates the actual control behavior at smaller time intervals.

[0152] The reason for using this hierarchical model structure is that traditional MPC models need to consider both global path objectives (such as reaching the target point) and local obstacle avoidance requirements (such as avoiding dynamic obstacles). When considering local obstacle avoidance, the computational load is significant if the dynamic risk potential field is taken into account. This hierarchical predictive control model structure reduces the computational load for local obstacle avoidance (because the reference trajectory generated by the upper-level predictive control model reduces computational effort), while this reference trajectory also guides the UAV's path on medium- to long-term timescales, improving flight efficiency.

[0153] The following is in conjunction with the appendix Figure 2 The present application provides an exemplary system diagram for aircraft obstacle avoidance, illustrating some embodiments thereof.

[0154] Please see the appendix Figure 2 , Figure 2 This application provides a system diagram for obstacle avoidance of an aircraft, which may include: a binocular vision perception and geometric preprocessing module 210, a target detection and semantic segmentation module 220, a motion state estimation and short-term trajectory prediction module 230, a risk assessment and feasible space generation module 240, and a hierarchical model predictive control obstacle avoidance module 250.

[0155] Specifically, the binocular vision perception and geometric preprocessing module 210 acquires dense point cloud data through binocular image (i.e., original image) matching; during this process, geometric correction, disparity optimization, and noise filtering are performed. This module can adaptively adjust the matching resolution according to flight speed and onboard computing power to achieve low-latency spatial modeling. Unlike traditional binocular perception methods, this module introduces an adaptive matching resolution and dynamic confidence filtering mechanism in the disparity calculation stage, which can adaptively adjust the calculation accuracy and cost function according to the current flight state of the UAV, balancing perception accuracy and onboard computing real-time performance.

[0156] The object detection and semantic segmentation module 220 employs a lightweight convolutional network to recognize multiple types of obstacles and fuses semantic results with depth information to generate a "semantic point cloud". This module not only outputs obstacle types and boundaries but also provides semantic priors for the prediction module.

[0157] Motion state estimation and short-term trajectory prediction module 230: Based on a multi-frame historical state sequence, it uses a learning-driven model to predict the future position distribution of obstacles and outputs the predicted mean (i.e. the future motion trajectory) of each obstacle at several future time steps, thus realizing uncertainty perception prediction.

[0158] Risk assessment and feasible space generation module 240: Models the obstacle prediction distribution as a Gaussian risk field (i.e., spatiotemporal risk potential field) and calculates the collision probability of the UAV trajectory in the prediction time domain, providing a safe boundary (i.e., safe flight domain) for control optimization.

[0159] Hierarchical Model Predictive Control (MPC) Obstacle Avoidance Module 250: This module adopts a two-layer structure, namely the upper-layer predictive control model Top MPC: plans the global reference trajectory; the lower-layer predictive control model Bottom MPC: further optimizes the local trajectory under dynamic risk constraints in combination with the global reference trajectory, and generates the UAV control behavior sequence (i.e. obstacle avoidance trajectory).

[0160] It is understood that the specific implementation process of each of the above modules can refer to the method implementation examples provided above. To avoid repetition, detailed descriptions are omitted here.

[0161] As can be seen from the above embodiments of this application, this application uses binocular vision as the core perception method to construct dense depth and semantic spatial information in real time; it estimates the future trajectory distribution of obstacles through the motion state estimation and short-term trajectory prediction module 230, and introduces a risk assessment mechanism to construct a dynamic safety domain; then it introduces a hierarchical model predictive control (MPC) framework to incorporate risk constraints and energy consumption optimization into the rolling time domain optimization problem, and finally realizes the transformation from "perception-response obstacle avoidance" to "predictive decision-making obstacle avoidance", thereby achieving precise obstacle avoidance of UAVs.

[0162] Please refer to Figure 3 , Figure 3 The diagram shows a block diagram of an obstacle avoidance device for an aircraft according to some embodiments of this application. It should be understood that the obstacle avoidance device corresponds to the method embodiments described above and is capable of performing the various steps involved in the method embodiments described above. The specific functions of the obstacle avoidance device can be found in the description above, and detailed descriptions are appropriately omitted here to avoid repetition.

[0163] Figure 3 The aircraft obstacle avoidance device includes at least one software function module that can be stored in a memory or embedded in the aircraft obstacle avoidance device in the form of software or firmware. The aircraft obstacle avoidance device includes: an acquisition module 310, used to acquire the future motion trajectory of obstacles in the flight area of ​​the aircraft; a construction module 320, used to construct a spatiotemporal risk potential field based on the future motion trajectory and the flight state information of the aircraft at the current moment; wherein, the flight state information includes the aircraft position and the aircraft velocity; the spatiotemporal risk potential field characterizes the degree of risk of collision between the obstacle and the aircraft in physical space; the spatiotemporal risk potential field includes a safe flight space; a prediction module 330, used to predict the flight reference trajectory of the aircraft through the flight state information; and a correction module 340, used to correct the flight reference trajectory using the spatiotemporal risk potential field to obtain the obstacle avoidance trajectory of the aircraft.

[0164] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0165] Some embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can perform the operation of any of the methods corresponding to the methods provided in the above embodiments.

[0166] Some embodiments of this application also provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the above embodiments provided in the above embodiments.

[0167] like Figure 4 As shown, some embodiments of this application provide an electronic device 400, which includes a memory 410, a processor 420, and a computer program stored in the memory 410 and executable on the processor 420. When the processor 420 reads the program from the memory 410 via a bus 430 and executes the program, it can implement the methods of any of the above embodiments.

[0168] Processor 420 can process digital signals and may include various computing architectures. For example, it may be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 420 may be a microprocessor.

[0169] Memory 410 can be used to store instructions executed by processor 420 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 420 of this disclosure embodiment can be used to execute instructions in memory 410 to implement the methods shown above. Memory 410 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0170] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0171] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0172] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for obstacle avoidance by an aircraft, characterized in that, include: To obtain the future trajectory of obstacles in the flight area of ​​the aircraft; Based on the future trajectory and the flight status information of the aircraft at the current moment, a spatiotemporal risk potential field is constructed; wherein, the flight status information includes the aircraft position and the aircraft velocity; the spatiotemporal risk potential field characterizes the degree of risk of the obstacle colliding with the aircraft in physical space; the spatiotemporal risk potential field includes a safe flight space; The flight reference trajectory of the aircraft is predicted based on the flight status information; The flight reference trajectory is corrected using the spatiotemporal risk potential field to obtain the obstacle avoidance trajectory of the aircraft.

2. The method as described in claim 1, characterized in that, The acquisition of the future trajectory of obstacles in the flight area of ​​the aircraft includes: Acquire dense point cloud data corresponding to the original images collected in real time by the aircraft; wherein, the original images include a left image frame and a right image frame; The dense point cloud data is converted into a semantically enhanced point cloud; wherein the semantically enhanced point cloud includes the point cloud coordinates, semantic labels, and confidence scores of the obstacles; the semantic labels represent the categories of the obstacles; Based on the semantically enhanced point cloud, the obstacle is predicted, and the future motion trajectory is determined.

3. The method as described in claim 2, characterized in that, The acquisition of dense point cloud data corresponding to the original images collected in real time by the aircraft includes: The original image is geometrically corrected to obtain a corrected image pair; wherein the corrected image pair includes a left corrected image and a right corrected image; The disparity value of each pixel in the corrected image pair is calculated using the disparity cost function; The depth map is obtained by calculating the disparity values; The dense point cloud data is obtained by restoring the coordinates of the corrected depth map after the depth map correction.

4. The method as described in claim 3, characterized in that, The step of converting the dense point cloud data into a semantically enhanced point cloud includes: Obtain the pixel category probability map of any corrected image; wherein, the any corrected image is either a left corrected image or a right corrected image; Input any of the corrected images into the target detection network to obtain a set of target detection boxes; The pixels within each detection box in the target detection box set are back-projected onto the three-dimensional space of the dense point cloud data using the corrected depth map to obtain a local point cloud. After aligning the point coordinates in the local point cloud with the pixel coordinates in any of the corrected images, the pixel category probability map is projected onto the local point cloud to obtain the semantically enhanced point cloud.

5. The method according to any one of claims 2-4, characterized in that, The step of predicting obstacles based on the semantically enhanced point cloud and determining the future trajectory includes: The obstacles are clustered based on the semantic labels in the semantically enhanced point cloud to obtain the obstacle set at the current moment; Determine the obstacle state sequence of the obstacles in the obstacle set; Construct a spatiotemporal graph corresponding to the state sequence of the obstacles; The spatiotemporal graph is input into a spatiotemporal graph convolutional neural network to obtain the future motion trajectory.

6. The method according to any one of claims 1-4, characterized in that, The construction of a spatiotemporal risk potential field based on the future trajectory and the flight status information of the aircraft at the current moment includes: Using the future trajectory and the flight status information, calculate the relative position and relative velocity of the aircraft and any obstacle among all obstacles at at least one future moment; Calculate the collision factor between the aircraft and any obstacle based on the relative position and the relative velocity; The collision factors of all the obstacles are calculated to construct the spatiotemporal risk potential field.

7. The method according to any one of claims 1-4, characterized in that, The safe flight space was obtained in the following way: The region in the spatiotemporal risk potential field with a risk value less than a preset risk threshold is defined as the safe flight space.

8. The method according to any one of claims 1-4, characterized in that, The prediction of the aircraft's flight reference trajectory using the flight status information includes: The flight status information and the original planned route of the aircraft are input into the upper-level predictive control model to obtain the flight reference trajectory of the aircraft at at least one future moment; wherein, the flight reference trajectory includes flight position and flight parameters; the upper-level predictive control model includes optimization functions and flight constraints.

9. The method according to any one of claims 1-4, characterized in that, The step of correcting the flight reference trajectory using the spatiotemporal risk potential field to obtain the obstacle avoidance trajectory of the aircraft includes: The flight state information, the flight reference trajectory, and the spatiotemporal risk potential field are input into the lower-level predictive control model to obtain the obstacle avoidance trajectory; wherein, the lower-level predictive control model includes multiple optimization objective functions and flight constraints.

10. A device for obstacle avoidance of an aircraft, characterized in that, include: The acquisition module is used to acquire the future motion trajectories of obstacles in the flight area of ​​the aircraft; A construction module is used to construct a spatiotemporal risk potential field based on the future trajectory and the flight status information of the aircraft at the current moment; wherein, the flight status information includes the aircraft position and the aircraft velocity; the spatiotemporal risk potential field characterizes the degree of risk of the obstacle colliding with the aircraft in physical space; the spatiotemporal risk potential field includes a safe flight space; The prediction module is used to predict the flight reference trajectory of the aircraft based on the flight status information; The correction module is used to correct the flight reference trajectory using the spatiotemporal risk potential field to obtain the obstacle avoidance trajectory of the aircraft.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to perform the method as described in any one of claims 1-9.

12. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as described in any one of claims 1-9.

Images