Unmanned aerial vehicle autonomous obstacle avoidance and path planning method and system based on deep learning

Abstract

The invention provides an unmanned aerial vehicle autonomous obstacle avoidance and path planning method and system based on deep learning, and relates to the field of unmanned aerial vehicle control, and the method comprises the steps: obtaining position information and environment perception data, constructing a spatial-temporal feature matrix, extracting target motion and background feature vectors, and mapping the target motion and background feature vectors into a target-environment fusion feature field; calculating an accessibility matrix and a cost matrix to construct a track search space, generating a candidate track set and determining an optimal planned track; and performing segmented optimization on the planned track to obtain a continuous attitude sequence, and generating an adaptive control strategy. According to the invention, intelligent obstacle avoidance and efficient path planning of the unmanned aerial vehicle in a complex environment are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle control, in particular to an unmanned aerial vehicle autonomous obstacle avoidance and path planning method and system based on deep learning. BACKGROUND

[0002] With the rapid development of unmanned aerial vehicle technology, its application in complex environments such as inspection, logistics, and rescue is becoming increasingly widespread. Traditional path planning methods rely heavily on static mapping and heuristic search algorithms, making it difficult to deal with dynamic obstacles and complex terrain conditions. In particular, in urban low-altitude flight or forest crossing scenarios, there are problems of obstacle avoidance lag and discontinuous path, which restrict the safety and stability of unmanned aerial vehicle autonomous flight. In recent years, the advantages of deep learning in image recognition and spatio-temporal modeling are significant, providing a new approach to unmanned aerial vehicle environmental perception and path planning. However, existing methods based on deep learning focus on a single task of image recognition or trajectory prediction, lack joint modeling of dynamic targets and background environment, and have not yet formed an integrated solution framework for autonomous obstacle avoidance and global path planning, resulting in generated flight paths lacking real-time and continuity.

[0003] In addition, existing methods generally ignore the cooperative matching of planned flight paths and unmanned aerial vehicle motion control systems, and problems such as sharp changes in trajectory curvature and slow response of thrust allocation seriously affect trajectory tracking accuracy. Therefore, there is a need for an unmanned aerial vehicle autonomous obstacle avoidance and path planning method that can integrate target and environmental features, have global path planning capabilities, and support dynamic attitude control. SUMMARY

[0004] The embodiments of the present application provide an unmanned aerial vehicle autonomous obstacle avoidance and path planning method and system based on deep learning, which can solve the problems in the prior art.

[0005] In a first aspect, the embodiments of the present application provide an unmanned aerial vehicle autonomous obstacle avoidance and path planning method based on deep learning, comprising:

[0006] Obtaining position information and environmental perception data of the unmanned aerial vehicle;

[0007] Constructing a spatio-temporal feature matrix from the environmental perception data, extracting a target motion feature vector and a background feature vector in the spatio-temporal feature matrix, calculating a motion trajectory projection based on the target motion feature vector, constructing an environmental constraint boundary based on the background feature vector, and mapping the motion trajectory projection and the environmental constraint boundary to a three-dimensional grid space to obtain a target-environment fusion feature field;

[0008] The reachable matrix of the grid points is calculated based on the target-environment fusion feature field, the state transition cost between adjacent grid points is calculated to generate a cost matrix, the reachable matrix and the cost matrix are combined to construct a track search space, a candidate track set satisfying the motion constraint is generated in the track search space, and the optimal planning track is determined by calculating the cumulative cost of each track in the candidate track set.

[0009] According to the curvature change of the optimal planning track, a plurality of flight segments are divided, online trajectory optimization is performed on the flight segments to obtain a continuous attitude sequence, adaptive thrust distribution strategies and real-time attitude control strategies are generated based on the continuous attitude sequence combined with environmental disturbance compensation, and are combined into segmented execution instructions, and the unmanned aerial vehicle is controlled to complete the trajectory tracking of the flight segment according to the segmented execution instructions.

[0010] In an optional embodiment,

[0011] The spatiotemporal feature matrix is constructed from the environment perception data, and the target motion feature vector and the background feature vector are extracted from the spatiotemporal feature matrix, including:

[0012] A plurality of groups of environment data are obtained by sampling the environment perception data at a preset time interval, the groups of environment data are arranged in the time dimension to form a data stream, the data stream is filtered in the time domain and normalized in amplitude to obtain a time series data sequence, a multi-scale convolution kernel is used to extract features from the time series data sequence to obtain a multi-scale feature map, and the multi-scale feature map is fused to obtain a spatiotemporal feature matrix;

[0013] The data difference between adjacent time points in the spatiotemporal feature matrix is calculated and threshold segmentation is performed to obtain a motion target region and a background region;

[0014] The correlation coefficient matrix is obtained by calculating the cross-correlation coefficient of the data of adjacent time points in the motion target region, the motion direction vector is obtained by extracting the position offset corresponding to the maximum correlation from the correlation coefficient matrix, the target contour is obtained by performing morphological processing on the motion target region to extract the boundary coordinates, and the motion feature vector is obtained according to the centroid displacement of the target contour along the motion direction and the motion direction vector;

[0015] The depth map is obtained by extracting feature points from the background region and calculating the parallax, the depth information is obtained by bilateral filtering the depth map, the edge point set is obtained by performing edge detection on the background region, the edge information is obtained by connecting the edge point set through a double-threshold method, and the background feature vector is obtained by combining the depth information and the edge information after spatial alignment.

[0016] In an optional embodiment,

[0017] According to the target motion feature vector, a motion trajectory projection is calculated, and an environmental constraint boundary is constructed according to a background feature vector, the motion trajectory projection and the environmental constraint boundary are mapped to a three-dimensional grid space to obtain a target-environment fusion feature field, which includes:

[0018] According to the target motion feature vector, a state equation is established to calculate a target position change and a velocity change, a target future position sequence is predicted based on the target position change and the velocity change by using a recursive iteration method to obtain a target motion trajectory, and the target motion trajectory is projected into an observation space by coordinate transformation to obtain a motion trajectory projection;

[0019] Point cloud data is constructed using depth information in the background feature vector, and the point cloud data is segmented and clustered to obtain an environmental three-dimensional space structure, an obstacle boundary is obtained from the environmental three-dimensional space structure by extracting an obstacle contour, and a passable boundary is obtained by calculating a passable area, and the obstacle boundary and the passable boundary are combined to obtain an environmental constraint boundary;

[0020] The motion trajectory projection is divided into grid cells according to a preset resolution and mapped to a three-dimensional grid space to obtain trajectory grid features, and the environmental constraint boundary is divided into grid cells according to a preset resolution and mapped to a three-dimensional grid space to obtain boundary grid features;

[0021] The state consistency between adjacent grid cells of the trajectory grid features is calculated to obtain a motion continuity score, the spatial coverage and boundary integrity of the boundary grid features are calculated to obtain a structure integrity score, the motion continuity score and the structure integrity score are normalized to obtain a feature fusion weight, and the trajectory grid features and the boundary grid features are adaptively weighted and superimposed according to the feature fusion weight to obtain a target-environment fusion feature field.

[0022] In an optional embodiment,

[0023] Based on the target-environment fusion feature field, a grid point reachability matrix is calculated, a state transition cost between adjacent grid points is calculated to generate a cost matrix, and the reachability matrix and the cost matrix are combined to construct a track search space, which includes:

[0024] The eigenvalue gradient of each grid point in the target-environment fusion feature field is calculated, the eigenvalue gradient is decomposed in a three-dimensional space to obtain gradient components, the gradient main direction is determined based on the gradient components, and the gradient projection value of the grid point is calculated according to the gradient main direction, the adjacent grid points are divided into multiple gradient descent domains based on the gradient projection value, and the adjacent points with eigenvalues smaller than the current grid point are selected as reachable candidate points in the multiple gradient descent domains;

[0025] Determine the detection step size based on the rate of change of the eigenvalues ​​in the area where the line connecting the current grid point and the reachable candidate point is located. Use the detection step size to perform segment-by-segment detection on the line to determine whether it intersects with the obstacle boundary. Check the continuity of the eigenvalues ​​on the line. Determine the reachable candidate points that do not intersect with obstacles and whose eigenvalues ​​change continuously as reachable points. Establish a reachability matrix based on the reachable point distribution of each grid point.

[0026] Calculating the ratio of the number of reachable points of each grid point in the reachability matrix to the total number of adjacent points to obtain the reachability, performing hierarchical weighting on the reachability according to the level of the gradient descent domain in which the grid point is located to obtain a weight coefficient, combining the weight coefficient with the distance between grid points, the heading change, and the characteristic field change rate to obtain a state transition cost, and constructing a state transition cost matrix;

[0027] The reachability matrix and the state transition cost matrix are combined to construct a track search space.

[0028] In an optional embodiment,

[0029] Generate a set of candidate tracks that meet the motion constraints in the track search space, and determine the optimal planned track by calculating the cumulative cost of each track in the candidate track set.

[0030] In the track search space, the weight coefficients of the grid points are normalized to obtain the basic transition probability, the state transition probability distribution is calculated based on the basic transition probability and the main direction of the gradient, and the initial sampling node set is generated in the gradient descent domain at each level;

[0031] The node search radius is determined according to the level of the gradient descent domain where the sampling node is located. A neighbor search is performed within the search radius to obtain a set of connectable nodes. Based on the reachability matrix, whether the connectable nodes meet the motion constraints is determined. Nodes that meet the motion constraints are connected and assigned state transition costs to construct a candidate track search tree.

[0032] Performing motion constraint verification on the paths in the candidate track search tree, extracting path nodes that satisfy steering angle constraints and steering angular acceleration constraints as a control point sequence, and performing trajectory smoothing on the control point sequence to generate a candidate track set;

[0033] The hierarchical cost weights are set based on the state transition cost and the gradient descent domain level, the cumulative cost of each track in the candidate track set is calculated, the track with the smallest cumulative cost is selected as the current optimal track, local sampling is performed with the control point of the optimal track as the center to obtain a new control point sequence, the new control point sequence is trajectory smoothed to generate new candidate tracks, the cumulative cost of the new candidate tracks is calculated and the optimal track is updated, and local optimization is iteratively performed until the cumulative cost converges to obtain the optimal planned track.

[0034] In an alternative embodiment,

[0035] The method for dividing the flight segments according to the curvature variation of the optimal planned trajectory, and performing online trajectory optimization on the flight segments to obtain a continuous attitude sequence comprises:

[0036] Obtaining the curvature variation values of adjacent trajectory points on the optimal planned trajectory, determining the trajectory points with curvature variation values greater than a preset curvature threshold as curvature mutation points, and dividing the optimal planned trajectory into multiple flight segments according to the curvature mutation points;

[0037] Establishing a sliding time window with a fixed length for each flight segment, setting the initial position of the sliding time window at the starting end of the flight segment, and in the sliding time window, determining the expected overload instruction range and the expected roll rate range of the control points based on the curvature gradient sequence of the current position, and generating multiple groups of candidate control input sequences;

[0038] Selecting a control input sequence with the minimum curvature variation that satisfies the flight dynamics constraints from the multiple groups of candidate control input sequences as the optimal control input, and generating a continuous attitude sequence for the current time window according to the optimal control input;

[0039] Taking the end state of the continuous attitude sequence as the initial state of the next sliding time window, sliding the sliding time window forward and repeating the optimization process until the trajectory optimization of the current flight segment to be optimized is completed, and after completing the trajectory optimization of all flight segments, combining the continuous attitude sequences of the optimized flight segments to obtain a final continuous attitude sequence.

[0040] In an alternative embodiment,

[0041] Generating an adaptive thrust allocation strategy and a real-time attitude control strategy based on the continuous attitude sequence combined with the environmental disturbance compensation, and combining them into a segmented execution instruction, and controlling the UAV to complete the trajectory tracking of the flight segment according to the segmented execution instruction comprises:

[0042] Calculating the expected attitude instruction and the expected trajectory information based on the continuous attitude sequence, collecting the current flight state and the environmental disturbance information of the UAV, and comparing the current flight state with the expected attitude instruction to obtain an attitude deviation value;

[0043] Calculating the thrust compensation amount according to the attitude deviation value, adaptively adjusting the thrust compensation amount combined with the environmental disturbance information, generating an adaptive thrust allocation strategy, and generating a real-time attitude control strategy according to the expected attitude instruction and the attitude deviation value;

[0044] Dividing the flight segment into multiple continuous control intervals according to the feature points of the expected trajectory information, and combining the adaptive thrust allocation strategy and the real-time attitude control strategy to form a segmented execution instruction;

[0045] A corresponding control parameter is configured in each control interval, and a smooth transition function is used at the boundary point of adjacent control intervals to realize continuous switching of the segmented execution instruction, and the unmanned aerial vehicle is controlled to complete trajectory tracking of the flight segment according to the switched segmented execution instruction.

[0046] The second aspect of the embodiment of the application provides an unmanned aerial vehicle autonomous obstacle avoidance and path planning system based on deep learning, comprising:

[0047] The first unit is configured to acquire position information and environment perception data of the unmanned aerial vehicle.

[0048] The second unit is configured to construct a space-time feature matrix for the environment perception data, extract a target motion feature vector and a background feature vector in the space-time feature matrix, calculate a motion trajectory projection according to the target motion feature vector, construct an environment constraint boundary according to the background feature vector, and map the motion trajectory projection and the environment constraint boundary to a three-dimensional grid space to obtain a target-environment fusion feature field.

[0049] The third unit is configured to calculate a grid point reachability matrix based on the target-environment fusion feature field, calculate a state transition cost between adjacent grid points to generate a cost matrix, combine the reachability matrix and the cost matrix to construct a track search space, generate a candidate track set satisfying a motion constraint in the track search space, and determine an optimal planning track by calculating cumulative costs of tracks in the candidate track set.

[0050] The fourth unit is configured to divide a plurality of flight segments according to curvature changes of the optimal planning track, perform online trajectory optimization on the flight segments to obtain a continuous attitude sequence, generate an adaptive thrust distribution strategy and a real-time attitude control strategy based on the continuous attitude sequence combined with environment disturbance compensation, and combine the adaptive thrust distribution strategy and the real-time attitude control strategy into a segmented execution instruction, and control the unmanned aerial vehicle to complete trajectory tracking of the flight segment according to the segmented execution instruction.

[0051] The third aspect of the embodiment of the application provides an electronic device, comprising:

[0052] A processor;

[0053] A memory for storing processor-executable instructions;

[0054] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0055] The fourth aspect of the embodiment of the application provides a computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions are executed by a processor to implement the method described above.

[0056] In the embodiment, the target motion feature and the background feature are extracted by constructing a space-time feature matrix, the motion trajectory projection and the environment constraint boundary are mapped to a three-dimensional grid space to form a fusion feature field, the accurate perception and trajectory prediction of the mobile obstacle by the unmanned aerial vehicle in the dynamic environment are realized, and the accuracy and real-time performance of the obstacle avoidance decision are improved. The grid point accessibility and state transition cost are calculated based on the fusion feature field, the trajectory search space is constructed, the candidate trajectory set satisfying the motion constraint is generated, the optimal planning trajectory is determined through cumulative cost evaluation, the unmanned aerial vehicle can adaptively plan a safe and efficient flight path in the complex environment, and the local optimum and path non-smoothness problems in the traditional method are avoided. The optimal planning trajectory is divided into multiple flight segments and is subjected to online trajectory optimization, adaptive thrust allocation and real-time attitude control strategies are generated in combination with the environment disturbance compensation, the accurate trajectory tracking of the unmanned aerial vehicle in the uncertain environment is realized, the stability and robustness in the flight process are improved, and the energy consumption is reduced and the endurance time is prolonged. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 A flowchart of the method for autonomous obstacle avoidance and path planning of the unmanned aerial vehicle based on deep learning in the embodiment of the present application is shown in

[0058] Figure 2 A flowchart of the trajectory planning of the unmanned aerial vehicle in the embodiment of the present application is shown in DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in a clear and complete manner with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0060] The technical solutions of the present application will be described in detail below with reference to the specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0061] Figure 1 A flowchart of the method for autonomous obstacle avoidance and path planning of the unmanned aerial vehicle based on deep learning in the embodiment of the present application is shown in Figure 1 The method comprises the following steps:

[0062] acquiring position information and environment perception data of the unmanned aerial vehicle;

[0063] The spatio-temporal feature matrix is constructed from the environment perception data, target motion feature vectors and background feature vectors are extracted from the spatio-temporal feature matrix, a motion trajectory projection is calculated according to the target motion feature vectors, and an environment constraint boundary is constructed according to the background feature vectors, the target-environment fusion feature field is obtained by mapping the motion trajectory projection and the environment constraint boundary to a three-dimensional grid space;

[0064] A grid point reachability matrix is calculated based on the target-environment fusion feature field, a state transition cost matrix is generated by calculating the state transition cost between adjacent grid points, a reachability matrix and a cost matrix are combined to construct a trajectory search space, a candidate trajectory set satisfying the motion constraint is generated in the trajectory search space, and an optimal planning trajectory is determined by calculating the cumulative cost of each trajectory in the candidate trajectory set;

[0065] According to the curvature change of the optimal planning trajectory, a plurality of flight segments are divided, a continuous attitude sequence is obtained by performing online trajectory optimization on the flight segments, an adaptive thrust allocation strategy and a real-time attitude control strategy are generated based on the continuous attitude sequence combined with environmental disturbance compensation, and are combined into a segmented execution instruction, and the unmanned aerial vehicle is controlled to complete trajectory tracking of the flight segment according to the segmented execution instruction.

[0066] In an optional implementation, constructing a spatio-temporal feature matrix from the environment perception data, and extracting target motion feature vectors and background feature vectors from the spatio-temporal feature matrix include:

[0067] A plurality of groups of environment data are obtained by sampling the environment perception data at a preset time interval, the groups of environment data are arranged in the time dimension to form a data stream, time domain filtering and amplitude normalization are performed on the data stream to obtain a time series data sequence, multi-scale feature extraction is performed on the time series data sequence using a multi-scale convolution kernel to obtain a multi-scale feature map, and the multi-scale feature map is fused to obtain a spatio-temporal feature matrix;

[0068] The data difference between adjacent time points in the spatio-temporal feature matrix is calculated and threshold segmentation is performed to obtain a motion target region and a background region;

[0069] A correlation coefficient matrix is obtained by calculating the cross-correlation coefficient of the data between adjacent time points in the motion target region, a motion direction vector is obtained by extracting the position offset corresponding to the maximum correlation from the correlation coefficient matrix, a target contour is obtained by performing morphological processing on the motion target region to extract boundary coordinates, and a motion feature vector is obtained according to the centroid displacement of the target contour along the motion direction and the motion direction vector;

[0070] A depth map is obtained by extracting feature points from the background region and calculating parallax, depth information is obtained by performing bilateral filtering on the depth map, an edge point set is obtained by performing edge detection on the background region, edge information is obtained by connecting the edge point set through a double-threshold method, and a background feature vector is obtained by combining the depth information and the edge information after spatial alignment.

[0071] In a specific implementation, the environment perception data can come from raw data collected by various sensing devices such as radar, lidar, vision sensors, etc. First, the environment perception data is sampled according to a preset time interval of 10 milliseconds to obtain multiple groups of environment data. For example, in one implementation process, environment data is collected every 10 milliseconds, lasting for 5 seconds, and a total of 500 environment data samples are obtained. These environment data are arranged in the time dimension to form a continuous data stream. A band-pass filter is applied to the data stream for time domain filtering, with the low cutoff frequency set to 0.5 Hz and the high cutoff frequency set to 20 Hz to filter out environmental noise and irrelevant signal components. Then, the filtered data is subjected to amplitude normalization processing to scale the data range to the interval [-1, 1] to obtain a time series data sequence.

[0072] For the normalized time series data sequence, multi-scale convolution kernels are used for feature extraction. In specific implementation, three different size convolution kernels are designed: 3x3, 5x5 and 7x7. Small size convolution kernels are used to capture local fine features, and large size convolution kernels are used to capture more extensive context features. For each convolution kernel, 16 channels are set respectively, the convolution step is 1, and the edge uses zero padding to keep the feature map size consistent. Through convolution operations of the three different size convolution kernels on the time series data sequence, three feature maps are obtained respectively. These three feature maps are fused by channel concatenation and reduced by 1x1 convolution to finally obtain a spatio-temporal feature matrix with dimensions MxNxK, where M and N represent the spatial dimensions of the feature map respectively, and K represents the feature dimension, which can be set to 128x128x64 in actual application.

[0073] In the constructed spatio-temporal feature matrix, the data difference between adjacent time points is calculated to identify the moving target region and the background region. First, the Euclidean distance between adjacent two frames in the spatio-temporal feature matrix is calculated to obtain a difference matrix D. In the difference matrix, the larger the value, the more obvious the change at that position. An adaptive threshold segmentation method is applied to the difference matrix, and the threshold T is set to the mean value of the difference matrix plus 2 times the standard deviation. The region with a difference value greater than the threshold T is marked as the moving target region, and the region with a difference value less than or equal to the threshold T is marked as the background region. In one example, the mean value of the difference matrix is 0.15, and the standard deviation is 0.08, so the threshold T is calculated as 0.31.

[0074] For the identified moving target region, the cross-correlation coefficient is calculated for the data of adjacent time, and a correlation coefficient matrix R is constructed. In a 64x64 pixel size moving target region, the cross-correlation coefficient of the target region at time t and time t+1 is calculated by a sliding window method. The window size is set to 16x16 pixels, and the step size is 4 pixels. From the calculated correlation coefficient matrix, the position offset (Δx, Δy) corresponding to the maximum correlation is found, which represents the motion direction vector of the target. For example, if the maximum correlation coefficient appears at the offset (8, 12), it indicates that the target moves 8 units in the x direction and 12 units in the y direction.

[0075] Morphological processing is applied to the moving target region to extract the target contour. First, the target region is binarized, and then the open operation (erosion followed by dilation) is applied to remove noise points, and the structure element is selected as a 3x3 rectangular kernel. Then the close operation (dilation followed by erosion) is applied to fill the internal cavities of the target, and the structure element is selected as a 5x5 rectangular kernel. After completing the morphological processing, the contour tracking algorithm is used to extract the target boundary coordinate set to obtain the target contour. The centroid coordinates (cx, cy) of the target contour are calculated, and the motion direction vector (Δx, Δy) obtained previously is combined to construct the motion feature vector, represented as (cx, cy, Δx, Δy, v), where v is the displacement size, calculated as (Δx 2 +Δy 2 ) 1 / 2 .

[0076] For the background region, feature points are extracted and the depth map is calculated. The corner detection algorithm is used to identify feature points in the background region, and the detection threshold is set to 0.01, with a maximum of 1000 feature points. For stereo vision data, the feature points in the left and right images are matched to calculate the disparity, and the disparity search range is set to 0 to 64 pixels. According to the camera calibration parameters and the calculated disparity value, the depth value is converted to generate the original depth map. In order to improve the quality of the depth map, bilateral filtering is applied to the depth map, with a spatial kernel size of 5x5, a spatial standard deviation of 3.0, and an intensity standard deviation of 0.1. Bilateral filtering can smooth the depth map while preserving edge information, reducing noise effects.

[0077] Edge detection is performed on the background region to supplement the environmental structure information. First, the gradient amplitude and direction of the background region are calculated, and the gradient threshold is set to 1.5 times the average gradient of the background region. The edge points are connected by a double threshold method, with a high threshold of 0.2 and a low threshold of 0.1. Points higher than the high threshold are directly marked as edge points, points lower than the low threshold are directly discarded, and points between the two thresholds are only marked as edge points when they are adjacent to the already determined edge points. In this way, a set of continuous edge points is obtained, forming the edge information.

[0078] Finally, the depth information and the edge information are combined by spatial alignment to obtain the background feature vector. The spatial alignment is realized by pixel-level mapping, ensuring that the depth map and the edge map are one-to-one corresponding in spatial position. The combined background feature vector is represented as (x, y, d, e), where (x, y) is the pixel coordinate, d is the depth value, and e is the edge intensity. For a background area of 128x128 pixels in practical application, a background feature description containing 16,384 four-dimensional feature vectors is constructed.

[0079] In the embodiment, the spatiotemporal feature matrix extracted by multi-scale convolution effectively captures the temporal variation and local features in the environmental perception data, improving the recognition accuracy of the moving target. The motion feature vector constructed by cross-correlation and morphological processing can accurately depict the motion direction and contour shape of the target, improving the robustness of path prediction. The depth information of the background area is extracted by disparity calculation and bilateral filtering, and the structural information is extracted by edge detection, ensuring that the background feature vector has complete geometric constraints and boundary characteristics, which helps to construct the environmental model in the real scene and provides accurate input for subsequent path planning and obstacle avoidance strategies.

[0080] In an alternative embodiment, a motion trajectory projection is calculated according to the target motion feature vector, an environmental constraint boundary is constructed according to the background feature vector, and the motion trajectory projection and the environmental constraint boundary are mapped to a three-dimensional grid space to obtain a target-environment fusion feature field, including:

[0081] A state equation is established according to the target motion feature vector to calculate the target position change and velocity change, a recursive iteration method is used based on the target position change and velocity change to predict the target future position sequence to obtain the target motion trajectory, and the motion trajectory is projected to the observation space through coordinate transformation to obtain the motion trajectory projection;

[0082] Point cloud data is constructed using the depth information in the background feature vector, and the point cloud data is segmented and clustered to obtain the environmental three-dimensional spatial structure. The obstacle boundary is obtained by extracting the obstacle contour from the environmental three-dimensional spatial structure, and the passable boundary is obtained by calculating the passable area. The obstacle boundary and the passable boundary are combined to obtain the environmental constraint boundary.

[0083] The motion trajectory projection is divided into grid cells according to a predetermined resolution and mapped to a three-dimensional grid space to obtain trajectory grid features, and the environmental constraint boundary is divided into grid cells according to a predetermined resolution and mapped to a three-dimensional grid space to obtain boundary grid features.

[0084] The state consistency between adjacent grid cells of the trajectory grid feature is calculated to obtain a motion continuity score, the spatial coverage and boundary integrity of the boundary grid feature are calculated to obtain a structure integrity score, the motion continuity score and the structure integrity score are normalized to obtain a feature fusion weight, and the trajectory grid feature and the boundary grid feature are adaptively weighted and superimposed according to the feature fusion weight to obtain a target-environment fusion feature field.

[0085] Exemplarily, after the system obtains the target motion feature vector, a state equation describing the motion state of the target is established. The state equation includes the position, velocity and change relationship of the target. Assuming that the current position of the target is (10, 15, 0) m, the velocity is (2, 1, 0) m / s, and the acceleration is (0.1, 0.2, 0) m / s 2 , a linear recursive method is used to calculate the target position sequence in the next 5 seconds with a time step of 0.1 second. Specifically, for each time point t+1, the position update formula is: the current position plus the velocity multiplied by the time step, plus the acceleration multiplied by half the square of the time step; the velocity update formula is: the current velocity plus the acceleration multiplied by the time step. After 50 iterations of calculation, a target future motion trajectory containing 51 points is obtained, for example, the position of the 10th point is (12.5, 16.7, 0) meters. Subsequently, the trajectory is transformed from the target coordinate system to the observation coordinate system, assuming that the transformation matrix parameters are a rotation angle of 30 degrees and a translation vector of (5, 8, 0) meters, and the motion trajectory projection in the observation space is obtained through coordinate transformation.

[0086] The background feature vector contains depth information with a resolution of 640x480 pixels. Point cloud data is generated using these depth values, with each pixel corresponding to a three-dimensional space point, totaling 307200 points. The region growing algorithm based on Euclidean distance is used to segment these point cloud data, with a clustering distance threshold of 0.05 meters and a minimum number of clustering points of 100. After segmentation and clustering, the main structures in the environment are obtained, such as the ground, walls and obstacles. The ground point cloud set contains about 150000 points, located in the plane area with z coordinate close to 0; the wall point cloud set contains about 80000 points, mainly distributed in the x-z and y-z planes; the obstacle point cloud set contains several independent clusters, such as tables (about 5000 points) and chairs (about 3000 points). The obstacle boundary is extracted from the point cloud clustering, and the specific method is to calculate the convex hull or bounding box of each cluster. For example, the bounding box of the table is a cuboid from (2, 3, 0) to (3, 4, 0.8) meters. The passable region, i.e. the passable boundary, is calculated by the complement of the obstacle boundary. The obstacle boundary is marked as a prohibited passable region, and the passable boundary is marked as a permitted passable region, and the environment constraint boundary is formed by combining them.

[0087] The motion trajectory projection is divided into grid cells according to a preset resolution of 0.1 meters and mapped into a three-dimensional grid space with a size of 100x100x10. For each point on the trajectory, its corresponding grid index is calculated, such as the position (12.5, 16.7, 0) corresponding to the grid index (125, 167, 0). To represent the continuity of the trajectory, the grid cells passed by the trajectory are assigned a value of 1, and the rest are 0, forming a trajectory grid feature. Similarly, the environmental constraint boundary is divided into grid cells according to the same resolution and mapped into the same size of three-dimensional grid space, and the obstacle boundary grid is assigned a value of -1 and the passable boundary grid is assigned a value of 0.5, forming a boundary grid feature.

[0088] The consistency of adjacent grid cells in the trajectory grid feature is calculated to evaluate the motion continuity. For each trajectory grid cell, check its 26 adjacent grids, if the adjacent grid is also part of the trajectory, increase the continuity count. The total continuity count of the trajectory is finally obtained as 47 (for 50 segments of trajectory), and the continuity ratio is 47 / 50=0.94, i.e. the motion continuity score. At the same time, the spatial coverage of the boundary grid feature is calculated, i.e. the ratio of the number of non-zero grid cells to the total number of grid cells, for example 35000 / 100000=0.35; the boundary integrity is calculated, i.e. the degree of boundary closure, which is assumed to be 0.88. The average value of the spatial coverage and the boundary integrity is 0.615, which is taken as the structure integrity score.

[0089] The motion continuity score and the structure integrity score are normalized by Min-Max normalization, and the normalized motion continuity score is 0.9 and the structure integrity score is 0.8. Based on the two scores, the feature fusion weight is calculated, the trajectory grid feature weight is 0.9 / (0.9+0.8)=0.529, and the boundary grid feature weight is 0.8 / (0.9+0.8)=0.471. Finally, according to these weights, the trajectory grid feature and the boundary grid feature are weighted and superimposed, such as for grid index (125, 167, 0), the fusion feature value is 0.529x1+0.471x0.5=0.765. Similar calculations are performed for all grid cells, and finally a fusion feature field representing the target motion trajectory and environmental constraints is obtained, which reflects the target future possible motion path and contains the obstacle and passable area information in the environment.

[0090] In the embodiment, the target motion trajectory constructed by the state equation and the recursive prediction can dynamically reflect the future motion trend of the target and be projected to the observation space to enhance the spatial correlation of the trajectory. Meanwhile, the environment constraint boundary constructed by the depth point cloud can accurately depict the obstacle shape and the passing area, and provide real and effective spatial restrictions for path planning. The motion trajectory and the environment boundary are mapped to the three-dimensional grid, and the feature fusion weight is calculated to balance the evaluation of the motion continuity and the environment structure integrity. The fusion feature field is constructed by the adaptive weighting method to effectively enhance the cooperative understanding of the dynamic target behavior and the static environment constraint, and provide accurate and structured prior support for the generation of subsequent feasible trajectories.

[0091] In an optional implementation, a grid point reachability matrix is calculated based on the target-environment fusion feature field, a state transition cost matrix is generated by calculating the state transition cost between adjacent grid points, and a trajectory search space is constructed by combining the reachability matrix and the cost matrix, including:

[0092] A feature value gradient of each grid point in the target-environment fusion feature field is calculated, the feature value gradient is decomposed in a three-dimensional space to obtain gradient components, a gradient main direction is determined based on the gradient components, a gradient projection value of the grid point is calculated according to the gradient main direction, adjacent grid points are divided into multi-level gradient descent domains based on the gradient projection value, and adjacent points with a feature value smaller than that of the current grid point are selected as reachable candidate points in the multi-level gradient descent domains;

[0093] A detection step is determined according to the feature value change rate of a region where a line connecting the current grid point and the reachable candidate point is located, the line is detected segment by segment using the detection step to determine whether the line intersects with the obstacle boundary, the continuity of the feature value on the line is checked, the reachable candidate points that do not intersect with the obstacle and have a continuous change in the feature value are determined as reachable points, and a reachability matrix is established according to the distribution of the reachable points of each grid point.

[0094] A reachability degree is obtained by calculating the proportion of the number of reachable points of each grid point in the reachability matrix in the total number of adjacent points, a weight coefficient is obtained by grading and weighting the reachability degree according to the level of the gradient descent domain where the grid point is located, a state transition cost is obtained by combining the weight coefficient with the distance between grid points, the change in the heading direction, and the feature field change rate, and a state transition cost matrix is constructed.

[0095] The reachability matrix and the state transition cost matrix are combined to construct a trajectory search space.

[0096] In this embodiment, the method of constructing the track search space first calculates the gradient of the feature value of each grid point in the target-environment fusion feature field. The feature field is a three-dimensional space representation composed of obstacles and targets, where each grid point contains feature value information. By calculating the difference between the feature values of the current grid point and its adjacent points, the rate of change of the feature value in the x, y, and z directions is determined, forming a gradient vector. For example, for a grid point with coordinates (10, 15, 20), the feature value is 0.75, and by comparing with the feature values of adjacent points, the gradient vector is calculated as (0.05, 0.12, -0.03).

[0097] The gradient components are obtained by decomposing the gradient in three-dimensional space. For the above gradient vector, the x-direction component is 0.05, the y-direction component is 0.12, and the z-direction component is -0.03. Based on these components, the gradient main direction is determined, which is the direction with the largest absolute value of the gradient component. In this example, the y-direction is the gradient main direction, with a component value of 0.12.

[0098] The gradient projection value of the grid point is calculated according to the gradient main direction. The position vector of each adjacent point relative to the current point is projected onto the gradient direction to obtain the projection value. For the adjacent point located at (11, 15, 20), the position vector is (1, 0, 0), and the projection value onto the gradient direction is 0.05 x 1 + 0.12 x 0 + (-0.03) x 0 = 0.05.

[0099] Based on the gradient projection value, the adjacent grid points are divided into multiple levels of gradient descent domains. According to the size of the projection value, the adjacent points are divided into strong gradient descent domain, medium gradient descent domain and weak gradient descent domain. Points with projection values less than -0.1 are classified as strong gradient descent domain, points with projection values between -0.1 and -0.05 are classified as medium gradient descent domain, and points with projection values between -0.05 and 0 are classified as weak gradient descent domain. In the multi-level gradient descent domain, the adjacent points with feature values smaller than the current grid point are selected as reachable candidate points. If the current point feature value is 0.75 and the adjacent point feature value is 0.70 and located in the gradient descent domain, the point is selected as a reachable candidate point.

[0100] Next, the detection step is determined according to the rate of change of the feature value in the region where the line connecting the current grid point and the reachable candidate point is located. If the rate of change of the feature value in the region is 0.02 per unit distance, the detection step is set to the inverse of the rate of change multiplied by a safety factor, i.e. 0.4, indicating that detection is performed every 0.4 unit distance. The detection step is used to detect the line segment by segment to determine whether it intersects with the obstacle boundary. Starting from the current point, every 0.4 unit distance is advanced, and the feature value at that position is checked. If the feature value suddenly increases by more than the threshold value 0.3, it indicates that the obstacle boundary may be crossed.

[0101] The continuity of the characteristic value on the connection line is checked. If the characteristic value of a point on the connection line is 1.5, and the characteristic values of the adjacent detection points before and after it are 0.6 and 0.7 respectively, the characteristic value jump exceeds the threshold value 0.3, and it is determined that the characteristic value is discontinuous. The reachable candidate point which does not intersect with the obstacle and the characteristic value changes continuously is determined as the reachable point. For example, the characteristic values on the connection line from the point (10, 15, 20) to the point (12, 16, 20) are 0.75, 0.72, 0.68, 0.65 respectively, which changes continuously and does not exceed the threshold value, and it is determined that (12, 16, 20) is a reachable point. The reachability matrix is established according to the distribution of the reachable points of each grid point, and the element value in the matrix is 1 indicating that the corresponding grid point is reachable, and 0 indicating that it is not reachable.

[0102] The proportion of the number of reachable points of each grid point in the reachability matrix to the total number of adjacent points is calculated to obtain the reachability degree. If a grid point has 26 adjacent points, of which 18 points are reachable points, the reachability degree of the point is 18 / 26 = 0.69. The reachability degree is graded and weighted according to the level of the gradient descent domain where the grid point is located to obtain the weight coefficient. For the points in the strong gradient descent domain, the weight coefficient is 0.6; for the points in the medium gradient descent domain, the weight coefficient is 0.8; for the points in the weak gradient descent domain, the weight coefficient is 1.0; and for the points in the non-gradient descent domain, the weight coefficient is 1.5.

[0103] The state transition cost is obtained by combining the weight coefficient, the distance between grid points, the heading change amount and the characteristic field change rate. For example, the distance from the current point (10, 15, 20) to the reachable point (12, 16, 20) is 2.24, the heading change amount is 0.1 radian, the characteristic field change rate is 0.05, the point is located in the medium gradient descent domain, and the weight coefficient is 0.8, then the state transition cost is 0.8 x (2.24 + 0.1 x 10 + 0.05 x 20) = 3.79. The state transition cost matrix is constructed, and the element value in the matrix is the state transition cost between the corresponding grid points, and the cost between the unreachable points is set to infinity.

[0104] Finally, the reachability matrix and the state transition cost matrix are combined to construct the track search space. The reachability matrix determines whether the grid points can be directly connected, and the state transition cost matrix provides the cost information of the connection. The search algorithm can find the optimal path from the starting point to the ending point in this space, avoiding obstacles and meeting the navigation constraint conditions. For example, the path search from the starting point (5, 5, 10) to the ending point (25, 25, 10), the feasible connection can be selected according to the reachability matrix, and the path with the minimum cost is selected through the state transition cost matrix, and finally the track point sequence is (5, 5, 10), (8, 7, 10), (12, 11, 11), (16, 15, 11), (20, 20, 10), (25, 25, 10), and the total cost is 42.5.

[0105] In the embodiment, the trend of terrain change and the passable direction can be accurately identified by spatial decomposition and direction projection of the feature value gradient of the grid points in the target-environment fusion feature field, and the construction of the multi-level gradient descent domain helps to dynamically adapt to the environment structure of different complexity. The detection step is set in combination with the feature value change rate, and the segment-by-segment connection detection is performed, so as to ensure that the reachable path is not intersected with the obstacle boundary in space and has good feature continuity, and the accuracy of reachability determination is improved. The reachability matrix constructed based on the distribution of reachable points reflects the connectivity characteristics of the path network, and the state transition cost matrix constructed by integrating factors such as gradient level, heading change and terrain complexity realizes the quantitative evaluation of the path passing difficulty, thereby forming a track search space with environmental adaptability and path optimization capability, which provides a solid foundation for generating high-quality autonomous tracks.

[0106] As shown in Figure 2 , the unmanned aerial vehicle trajectory planning process of the embodiment is shown.

[0107] In an optional implementation, a candidate track set satisfying the motion constraint is generated in the track search space, and the optimal planning track is determined by calculating the cumulative cost of each track in the candidate track set, including:

[0108] In the track search space, the weight coefficient of the grid point is normalized to obtain a basic transition probability, and the state transition probability distribution is calculated according to the basic transition probability and the gradient main direction, and an initial sampling node set is generated in each level of the gradient descent domain;

[0109] The node search radius is determined according to the level of the gradient descent domain where the sampling node is located, the near neighbor search is performed within the search radius to obtain a connectable node set, whether the connectable node satisfies the motion constraint is judged based on the reachability matrix, the nodes satisfying the motion constraint are connected and are given a state transition cost, and a candidate track search tree is constructed;

[0110] The path in the candidate track search tree is verified for the motion constraint, the path nodes satisfying the steering angle constraint and the steering angle acceleration constraint are extracted as a control point sequence, and the candidate track set is generated by trajectory smoothing of the control point sequence;

[0111] The hierarchical cost weight is set based on the state transition cost and the gradient descent domain level, the cumulative cost of each track in the candidate track set is calculated, the track with the minimum cumulative cost is selected as the current optimal track, the control points of the optimal track are taken as the center to obtain a new control point sequence, the new candidate track is generated by trajectory smoothing of the new control point sequence, the cumulative cost of the new candidate track is calculated and the optimal track is updated, and the local optimization is iteratively executed until the cumulative cost converges to obtain the optimal planning track.

[0112] Exemplarily, the weight coefficient of the grid point is normalized to obtain the basic transition probability in the track search space. Specifically, for each grid point (x, y) of the track search space, the weight coefficient w(x, y) is calculated according to the obstacle distance, terrain complexity, tactical threat and other factors of the point, and the basic transition probability p(x, y) is obtained by normalizing all the weight coefficients in the entire search space. For example, when the weight coefficient w(x, y) = 0.8 and the maximum weight coefficient in the search space is 1.0, the basic transition probability p(x, y) of the point is 0.8 / 1.0 = 0.8.

[0113] The state transition probability distribution is calculated according to the basic transition probability and the gradient main direction. The gradient main direction is determined by calculating the derivative of the basic transition probability of the grid point in each direction. For example, if the basic transition probability derivatives of the grid point (5, 3) in the east, south, west and north directions are 0.2, -0.1, -0.3 and 0.1 respectively, then the gradient main direction is the east direction. The state transition probability distribution is weighted according to the gradient main direction, the transition probability of the main direction increases, and the transition probability of the direction perpendicular to the main direction decreases. If the gradient main direction is east, the transition probability of the east direction can increase by 20%, the south and north directions decrease by 10%, and the west direction decreases by 30%.

[0114] An initial sampling node set is generated in each level gradient descent domain. The gradient descent domain is divided into multiple levels according to the size of the basic transition probability, for example, the region with a basic transition probability greater than 0.8 is a first level gradient descent domain, 0.6-0.8 is a second level, 0.4-0.6 is a third level, and so on. Different sampling densities are used in different levels of gradient descent domains, the first level domain has the highest sampling density, which can be set to one sampling point every 10 meters; the second level domain can be set to one sampling point every 20 meters; the third level domain can be set to one sampling point every 30 meters, and so on. In this way, the initial sampling node set is generated in the entire track search space.

[0115] The node search radius is determined according to the level of the gradient descent domain where the sampling node is located. For example, the search radius of the node in the first level domain is set to 50 meters, the search radius of the node in the second level domain is set to 40 meters, and the search radius of the node in the third level domain is set to 30 meters, and so on. The neighbor search is performed in the determined search radius range to obtain the connectable node set. For example, if node A is located in the first level gradient descent domain and the search radius is 50 meters, all the nodes within the search radius constitute the connectable node set of A.

[0116] The connectivity of nodes is determined based on the reachability matrix. The reachability matrix contains the minimum turning radius, maximum climbing / descending angle, maximum speed, and other constraint parameters of the mobile platform. For example, if the minimum turning radius of the mobile platform is 100 meters, the connection between two nodes is determined to be unreachable when the angle formed by the connection and the current heading of the node exceeds 30 degrees. For nodes that meet the motion constraints, a connection relationship is established and a state transition cost is assigned, and a candidate trajectory search tree is constructed. The state transition cost considers the weighted sum of distance, turning angle change, height change, and other factors. For example, the distance between two points is 200 meters, the turning angle change is 15 degrees, and the height change is 10 meters, then the state transition cost can be calculated as 200+15*5+10*10=375.

[0117] The paths in the candidate trajectory search tree are verified for motion constraints. The verification includes turning angle constraint and turning angle acceleration constraint. The turning angle constraint requires that the turning angle of adjacent path segments does not exceed the maximum allowed value, for example, 30 degrees; the turning angle acceleration constraint requires that the rate of change of the turning angle per unit time does not exceed the maximum allowed value, for example, 5 degrees per second. The nodes that meet these constraints are extracted as a control point sequence, for example, a path from the starting point to the ending point contains 10 nodes, after motion constraint verification, only nodes 1, 3, 5, 7, and 10 meet the constraints, then these 5 nodes constitute the control point sequence.

[0118] The control point sequence is smoothed to generate a candidate trajectory set. A cubic spline interpolation method is used to smooth the control point sequence, and multiple transition points are inserted between each two adjacent control points, so that the first and second derivatives of the trajectory at the control points are continuous, thereby ensuring the smoothness of the trajectory. For example, 5 transition points are inserted between the control point sequence (0, 0), (100, 100), (200, 150), (300, 200), and (400, 100), forming a smooth trajectory containing 25 points.

[0119] Based on the state transition cost and the gradient descent domain level, the hierarchical cost weight is set. The state transition cost weight in the first level gradient descent domain is the lowest, which can be set to 0.8; the second level domain is 1.0; the third level domain is 1.2, and so on. The cumulative cost of each trajectory in the candidate trajectory set is calculated, for example, the total cost of a certain trajectory through the first level domain is 300, the total cost through the second level domain is 200, and the total cost through the third level domain is 100, then the cumulative cost of the trajectory is 300*0.8+200*1.0+100*1.2=560. The trajectory with the minimum cumulative cost is selected as the current optimal trajectory.

[0120] A new control point sequence is obtained by local sampling with the control points of the optimal trajectory as the center. Five new points are randomly sampled within a 10-meter range around each control point, replacing the original control points to generate a new control point sequence. A new candidate trajectory is generated by trajectory smoothing on the new control point sequence, and the cumulative cost of the new candidate trajectory is calculated to update the optimal trajectory. For example, if the cumulative cost of the original optimal trajectory is 560 and the cumulative cost of the new trajectory obtained after local optimization is 530, the optimal trajectory is updated to the new trajectory. The local optimization is iteratively performed until the change in the cumulative cost is less than a preset threshold (e.g., 1%), at which point it is considered to have converged, and the final optimal planning trajectory is obtained.

[0121] In this embodiment, by constructing the basis transition probability and state transition probability distribution within the trajectory search space, the sampling nodes can be dynamically guided to gather in the gradient descent direction, improving the path search efficiency. The node search radius and motion constraint verification are set in layers to ensure that the generated trajectory meets the unmanned aerial vehicle maneuverability restrictions in terms of turning angle and acceleration. The multi-layer cost weight accumulation calculation of the candidate trajectory helps to comprehensively consider the path smoothness, passability and environmental complexity, achieving a balance between global optimality and local feasibility. The iterative optimization based on local sampling and trajectory smoothing further refines the optimal trajectory, significantly improves the path continuity and safety margin, and makes the final planning trajectory more suitable for efficient tracking execution in dynamic environments and under multiple constraint conditions.

[0122] In an optional implementation, dividing a plurality of flight segments according to the curvature change of the optimal planning trajectory, and performing online trajectory optimization on the flight segments to obtain a continuous attitude sequence includes:

[0123] Obtaining the curvature change value of adjacent trajectory points on the optimal planning trajectory, determining trajectory points with a curvature change value greater than a preset curvature threshold as curvature mutation points, and dividing the optimal planning trajectory into a plurality of flight segments based on the curvature mutation points;

[0124] Establishing a fixed-length sliding time window for each flight segment, setting the initial position of the sliding time window at the starting end of the flight segment, and within the sliding time window, determining the expected overload instruction range and the expected roll rate range of the control points based on the curvature gradient sequence of the current position, and generating a plurality of candidate control input sequences;

[0125] Selecting a control input sequence that satisfies the flight dynamics constraints and has the minimum curvature change from the plurality of candidate control input sequences as the optimal control input, and generating a continuous attitude sequence for the current time window based on the optimal control input;

[0126] The end state of the continuous attitude sequence is taken as the initial state of the next sliding time window, the sliding time window is slid forward and the optimization process is repeatedly performed until the trajectory optimization of the current flight segment to be optimized is completed. After the trajectory optimization of all flight segments is completed, the optimized continuous attitude sequences of the respective flight segments are combined to obtain the final continuous attitude sequence.

[0127] For example, before trajectory optimization, the optimal planning trajectory needs to be analyzed and processed first. The curvature change value of adjacent trajectory points on the optimal planning trajectory is obtained, and the calculation method is the absolute value of the difference between the curvature of the current trajectory point and the curvature of the previous trajectory point. In actual application, a preset curvature threshold of 0.05 rad / m can be set, and when the curvature change value of a certain trajectory point is greater than the threshold, it is marked as a curvature mutation point. For example, for a trajectory containing 100 trajectory points, it may be found through calculation that the curvature change values of the 25th point, the 47th point and the 78th point are 0.063 rad / m, 0.072 rad / m and 0.058 rad / m respectively, all of which exceed the preset threshold, so these three points are determined as curvature mutation points. With these curvature mutation points as boundaries, the entire optimal planning trajectory is divided into four flight segments: the first point to the 25th point, the 26th point to the 47th point, the 48th point to the 78th point and the 79th point to the 100th point.

[0128] For each flight segment divided, a sliding time window of fixed length is established for online trajectory optimization. The length of the sliding time window can be set to 2 seconds, and the sampling interval is 0.1 seconds, that is, each window contains 21 time points. Initially, the starting position of the sliding time window is set at the first trajectory point of the current flight segment to be optimized. Based on the curvature gradient sequence at the current position in the window, the expected overload command range and the expected roll rate range of the control point are determined.

[0129] When determining the control input range, according to the dynamic characteristics of the aircraft and the flight mission requirements, the overload command range can be set to 1g to 5g, where g represents the standard gravity acceleration (9.81 m / s 2 ) The roll rate range is set to -60° / s to 60° / s. A plurality of candidate control input sequences are generated in a discrete manner within the above ranges. For example, the overload command can be uniformly sampled to 9 values within the range of 1g to 5g: 1g, 1.5g, 2g, 2.5g, 3g, 3.5g, 4g, 4.5g and 5g; the roll rate is uniformly sampled to 7 values within the range of -60° / s to 60° / s: -60° / s, -40° / s, -20° / s, 0° / s, 20° / s, 40° / s and 60° / s. By combining these discrete values, 63 candidate control input sequences can be generated.

[0130] For each set of generated candidate control input sequences, forward simulation is performed using the flight dynamics model to calculate the corresponding flight trajectory and attitude change. Specifically, each set of candidate control input sequences (such as thrust, rudder angle, overload instruction, roll angle rate, etc.) is input as the driving signal of the model into the flight dynamics model. The flight dynamics model calculates the force and moment of the aircraft according to the rigid body kinematics and dynamics equations (such as Newton-Euler equations), and combines the current state (position, velocity, attitude, angular velocity, etc.) to perform numerical integration to obtain the motion state at the next time. By iterating step by step in time, the control input sequence is applied to the model to simulate the trajectory position and attitude change of the aircraft in continuous time to form a complete flight trajectory and attitude time sequence. During the simulation process, it is necessary to check whether the trajectory meets the flight dynamics constraints, including maximum overload constraint, maximum roll angle rate constraint, and attitude change continuity constraint, etc. For candidate sequences that meet the constraint conditions, calculate the curvature change value, and the specific method is to calculate the square sum of the curvature difference between adjacent points in the sequence. The smaller the curvature change value, the smoother the trajectory.

[0131] From the candidate control input sequences that meet the constraint conditions, select the sequence with the smallest curvature change value as the optimal control input. Assuming that in a certain time window, after calculation it is found that the control input sequence with an overload of 3g and a roll angle rate of 20° / s produces the smallest curvature change value of 0.012 rad 2 / m 2 Therefore, it is determined as the optimal control input of the current window. According to the optimal control input, the continuous attitude sequence in the current time window is generated through the flight dynamics model, including position, velocity, acceleration, heading angle, pitch angle, and roll angle parameters.

[0132] After completing the optimization of the current time window, the end state of the continuous attitude sequence is taken as the initial state of the next sliding time window. For example, if the end state of the current window after optimization is: position coordinates (1000m, 500m, 200m), velocity 300m / s, heading angle 45°, pitch angle 10°, roll angle 5°, these parameter values will be taken as the initial state of the next window. Move the sliding time window forward by a certain step, such as 0.5 seconds, and then repeat the above optimization process. Continue to perform sliding window optimization until the entire trajectory optimization of the current flight segment is completed. When optimizing to the end of the flight segment, it may be necessary to adjust the length of the last sliding window to ensure that the entire flight segment is covered. After completing the above optimization process for all flight segments, the continuous attitude sequences optimized for each flight segment are combined in turn to form a complete, smooth curvature change continuous attitude sequence as the final flight instruction of the aircraft.

[0133] Based on the above technical scheme, fine dynamic adjustment and continuous attitude control on the flight execution level of complex flight paths can be realized. By detecting and segmenting the curvature variation of the optimal planning flight path, the maneuver intensity variation region can be effectively identified and segmented optimization control is implemented, thereby enhancing the adaptability to high dynamic flight tasks. By using a sliding time window for online optimization by segment, combined with the dynamic adjustment of the control input range based on the curvature gradient characteristics, a more smooth and stable attitude change sequence can be generated under the premise of meeting the flight dynamics constraints, thereby reducing the load impact of sudden attitude commands on the flight system. The continuity and real-time performance of the whole flight segment attitude sequence are realized through window recursion optimization, which helps to improve the path tracking accuracy, flight stability and control response efficiency, and is particularly suitable for high dynamic obstacle avoidance flight and autonomous navigation execution tasks in complex terrain.

[0134] In an optional embodiment, adaptive thrust allocation strategies and real-time attitude control strategies are generated based on the continuous attitude sequence combined with environmental disturbance compensation, and are combined into segmented execution instructions to control the UAV to complete the trajectory tracking of the flight segment.

[0135] Based on the continuous attitude sequence, expected attitude commands and expected trajectory information are calculated, the current flight state and environmental disturbance information of the UAV are collected, and the attitude deviation value is obtained by comparing the current flight state with the expected attitude commands;

[0136] The thrust compensation amount is calculated according to the attitude deviation value, the thrust compensation amount is adaptively adjusted combined with the environmental disturbance information, the adaptive thrust allocation strategy is generated, and the real-time attitude control strategy is generated according to the expected attitude commands and the attitude deviation value;

[0137] According to the feature points of the expected trajectory information, the flight segment is divided into a plurality of continuous control intervals, and the adaptive thrust allocation strategy and the real-time attitude control strategy are combined to form segmented execution instructions;

[0138] The corresponding control parameters are configured in each control interval, and the continuous switching of the segmented execution instructions is realized by using a smooth transition function at the boundary points of adjacent control intervals, and the UAV is controlled to complete the trajectory tracking of the flight segment according to the switched segmented execution instructions.

[0139] In this embodiment, first, a continuous attitude sequence is obtained, and a third-order spline interpolation algorithm is used to process these discrete attitude points to generate a smooth continuous desired attitude curve, ensuring the coherence of the UAV movement. Based on this curve, the expected attitude command and expected trajectory information at each time point are calculated. Based on the generated smooth continuous attitude curve, the curve is discretized according to the preset sampling time interval to obtain the attitude angle values at each sampling time, including the roll angle, pitch angle and yaw angle, which are used as the expected attitude command at this time for real-time reference and control by the flight control system. At the same time, combined with the UAV flight speed information, the expected attitude angle is converted into a flight direction vector. By calculating the change of the flight direction vector between adjacent sampling time points and combining the speed, the corresponding spatial position change is derived, and thus the expected trajectory point sequence of the UAV in the three-dimensional space is cumulatively calculated. For example, in a certain flight task, the original 20 discrete attitude points can be interpolated to 100 smooth continuous attitude points, with a time interval of 50 milliseconds, covering the entire 5-second flight segment.

[0140] At the same time, the current flight state of the UAV is collected through accelerometers, gyroscopes and GPS sensors, including position coordinates, flight speed, attitude angles and angular rates. The environmental disturbance information is obtained through barometers, wind speed sensors and disturbance observers, among which the disturbance observer can estimate the wind speed, direction and airflow turbulence level. The system compares the current attitude angle with the expected attitude command to calculate the attitude deviation value, including the roll angle deviation, pitch angle deviation and yaw angle deviation.

[0141] Based on the above attitude deviation value, a nonlinear feedback control algorithm is used to calculate the thrust compensation. Specifically, the basic thrust demand is first calculated, and then the compensation coefficient is determined according to the size and rate of change of the attitude deviation. For example, when the roll angle deviation is detected to exceed 5 degrees, 10% of the thrust adjustment in the corresponding direction is increased according to the preset compensation rule. These basic compensation amounts are then combined with the environmental disturbance information for adaptive adjustment.

[0142] In the environmental disturbance compensation link, an adaptive disturbance compensation algorithm is used to process external disturbances. When the lateral wind speed is detected to reach 5 meters per second, 15% of the reverse thrust is added to the original thrust compensation to offset the wind effect. At the same time, to cope with sudden airflow changes, a fast response mechanism is also implemented, which can complete disturbance evaluation and thrust redistribution within 200 milliseconds. This adaptive mechanism ensures the stability of the UAV under different environmental conditions.

[0143] After combining the adaptive thrust allocation strategy and the real-time attitude control strategy, the flight segment needs to be divided into multiple continuous control intervals. The division is based on the characteristic points of the desired trajectory, such as sharp turning points, acceleration and deceleration points, hovering points, etc. For example, in a trajectory containing three stages of straight-line flight at constant speed, 90-degree turning and climbing, it will be divided into three control intervals, each of which is configured with different control parameters. The first interval uses a parameter combination that prioritizes speed control, the second interval uses a parameter combination that prioritizes attitude accuracy, and the third interval uses a parameter combination that prioritizes height stability.

[0144] To achieve smooth transition between control intervals, a weight function is used for gradual switching. At the boundary points of two adjacent control intervals, the control strategy will not be switched immediately, but the parameter weight will be adjusted gradually in a transition interval (usually 500 milliseconds). Specifically, a sigmoid function is used as the weight transition function to make the control quantity change smoothly over time, avoiding system oscillation caused by sudden changes. For example, when the UAV enters the turning area from straight-line flight, the weight of the turning control parameter will be gradually increased 250 milliseconds before the boundary, while the weight of the straight-line flight parameter will be gradually decreased. The entire transition process lasts 500 milliseconds, ensuring the smoothness of the UAV's action.

[0145] Finally, the segmented execution instructions processed according to the specific parameters of each control interval and the smooth transition function are used to generate real-time motor control signals. In each control cycle (usually 10 milliseconds), the current state is re-evaluated, the thrust allocation and attitude control strategy are updated, and the corresponding control instructions are executed to form a closed-loop control, ensuring that the UAV can accurately track the predetermined trajectory. Tests show that this method can make the UAV maintain a trajectory deviation of less than 0.5 meters and an attitude angle deviation of less than 3 degrees under 6-level wind conditions, significantly improving the accuracy and reliability of flight tasks.

[0146] Based on the above technical solutions, high-precision trajectory tracking and stable flight control of the UAV in a complex dynamic environment can be achieved. By comparing the continuous attitude sequence with the real-time flight state, the attitude deviation is accurately obtained, effectively guiding the adaptive adjustment of the thrust compensation amount, enhancing the response capability and control robustness to environmental disturbances. The segmented execution instructions subdivide the flight segment into multiple control intervals, and the smooth transition function is used to realize seamless switching of control parameters, ensuring the continuity and stability of flight control, and avoiding sudden changes and oscillations caused by control switching. The overall strategy integrates thrust allocation and attitude control, improving the dynamic adaptability and trajectory tracking accuracy of the UAV in a variable environment, and enhancing flight safety and task execution efficiency.

[0147] In a second aspect of the embodiments of the present application, a deep learning-based UAV autonomous obstacle avoidance and path planning system is provided, which comprises:

[0148] A first unit is configured to acquire position information and environment perception data of the UAV.

[0149] A second unit is configured to construct a space-time feature matrix from the environment perception data, extract a target motion feature vector and a background feature vector from the space-time feature matrix, calculate a motion trajectory projection based on the target motion feature vector, construct an environment constraint boundary based on the background feature vector, and map the motion trajectory projection and the environment constraint boundary to a three-dimensional grid space to obtain a target-environment fusion feature field.

[0150] A third unit is configured to calculate a grid point reachability matrix based on the target-environment fusion feature field, calculate a state transition cost between adjacent grid points to generate a cost matrix, combine the reachability matrix and the cost matrix to construct a trajectory search space, generate a candidate trajectory set that satisfies the motion constraint in the trajectory search space, and determine an optimal planning trajectory by calculating the cumulative cost of each trajectory in the candidate trajectory set.

[0151] A fourth unit is configured to divide a plurality of flight segments according to the curvature variation of the optimal planning trajectory, perform online trajectory optimization on the flight segments to obtain a continuous attitude sequence, generate an adaptive thrust distribution strategy and a real-time attitude control strategy based on the continuous attitude sequence combined with environmental disturbance compensation, and combine the adaptive thrust distribution strategy and the real-time attitude control strategy into a segmented execution instruction to control the UAV to complete trajectory tracking of the flight segments.

[0152] In a third aspect, an electronic device is provided, including:

[0153] a processor;

[0154] a memory for storing processor-executable instructions;

[0155] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0156] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0157] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer-readable storage medium having computer-readable program instructions loaded thereon, which are used to execute various aspects of the present application.

[0158] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. The deep learning-based autonomous obstacle avoidance and path planning method for UAVs is characterized by: include: Obtain the drone’s location information and environmental perception data; A spatiotemporal feature matrix is ​​constructed for environmental perception data. The target motion feature vector and background feature vector are extracted from the spatiotemporal feature matrix. The motion trajectory projection is calculated based on the target motion feature vector. The environmental constraint boundary is constructed based on the background feature vector. The motion trajectory projection and environmental constraint boundary are mapped to a three-dimensional grid space to obtain a target-environment fusion feature field. The grid point reachability matrix is ​​calculated based on the target-environment fusion feature field. The state transition costs between adjacent grid points are calculated to generate a cost matrix. The reachability matrix and the cost matrix are combined to construct a track search space. A set of candidate tracks that meet the motion constraints is generated in the track search space. The optimal planned track is determined by calculating the cumulative cost of each track in the candidate track set. The optimal planned trajectory is divided into multiple flight segments according to the curvature change, and the flight segments are optimized online to obtain a continuous attitude sequence. Based on the continuous attitude sequence combined with environmental disturbance compensation, an adaptive thrust allocation strategy and a real-time attitude control strategy are generated and combined into segmented execution instructions. According to the segmented execution instructions, the UAV is controlled to complete the trajectory tracking of the flight segment.

2. The method according to claim 1, characterized in that Constructing a spatiotemporal feature matrix for environmental perception data, extracting target motion feature vectors and background feature vectors from the spatiotemporal feature matrix includes: The environmental perception data is sampled at preset time intervals to obtain multiple sets of environmental data, each set of environmental data is arranged in the time dimension to form a data stream, the data stream is subjected to time domain filtering and amplitude normalization to obtain a time series data sequence, a multi-scale convolution kernel is used to extract features from the time series data sequence to obtain a multi-scale feature map, and the multi-scale feature map is fused to obtain a spatiotemporal feature matrix; Calculate the data differences at adjacent moments in the spatiotemporal feature matrix and perform threshold segmentation to obtain the moving target area and background area; Calculating the correlation coefficients of data at adjacent moments in the moving target region to obtain a correlation coefficient matrix, extracting the position offset corresponding to the maximum correlation from the correlation coefficient matrix to obtain a motion direction vector, performing morphological processing on the moving target region to extract boundary coordinates to obtain a target contour; and obtaining a motion feature vector based on the centroid displacement of the target contour along the motion direction and the motion direction vector; Feature points are extracted from the background area to calculate disparity to obtain a depth map, bilateral filtering is performed on the depth map to obtain depth information, edge detection is performed on the background area to obtain an edge point set, edge information is obtained by connecting them through a double threshold method, and the depth information and edge information are spatially aligned and combined to obtain a background feature vector.

3. The method according to claim 1, characterized in that The motion trajectory projection is calculated based on the target motion feature vector, and the environmental constraint boundary is constructed based on the background feature vector. The motion trajectory projection and the environmental constraint boundary are mapped to the three-dimensional grid space to obtain the target-environment fusion feature field, which includes: According to the target motion feature vector, a state equation is established to calculate the target position change and speed change. Based on the target position change and speed change, a recursive iterative method is used to predict the target's future position sequence to obtain the target motion trajectory, and the motion trajectory projection is obtained by projecting it into the observation space through coordinate transformation; Using the depth information in the background feature vector to construct point cloud data, segmenting and clustering the point cloud data to obtain a three-dimensional spatial structure of the environment, extracting obstacle contours from the three-dimensional spatial structure of the environment to obtain an obstacle boundary, calculating the traversable area to obtain a traversable boundary, and combining the obstacle boundary and the traversable boundary to obtain an environmental constraint boundary; Dividing the motion trajectory projection into grid units according to a preset resolution and mapping them to a three-dimensional grid space to obtain a trajectory grid feature, while dividing the environmental constraint boundary into grid units according to a preset resolution and mapping them to a three-dimensional grid space to obtain a boundary grid feature; The state consistency between adjacent grid cells of the trajectory grid feature is calculated to obtain a motion continuity score. The spatial coverage and boundary integrity of the boundary grid feature are calculated to obtain a structural integrity score. The motion continuity score and structural integrity score are normalized to obtain a feature fusion weight. The trajectory grid features and boundary grid features are adaptively weighted superimposed according to the feature fusion weight to obtain the target-environment fusion feature field.

4. The method according to claim 1, wherein The grid point reachability matrix is ​​calculated based on the target-environment fusion feature field, and the state transition cost between adjacent grid points is calculated to generate a cost matrix. The reachability matrix and the cost matrix are combined to construct the track search space, including: Calculate the eigenvalue gradient of each grid point in the target-environment fusion feature field, decompose the eigenvalue gradient in three-dimensional space to obtain gradient components, determine the main gradient direction based on the gradient components, and calculate the gradient projection value of the grid point based on the main gradient direction. Based on the gradient projection value, divide the adjacent grid points into a multi-level gradient descent domain, and select the adjacent points with eigenvalues ​​smaller than the current grid point in the multi-level gradient descent domain as reachable candidate points; Determine the detection step size based on the rate of change of the eigenvalues ​​in the area where the line connecting the current grid point and the reachable candidate point is located. Use the detection step size to perform segment-by-segment detection on the line to determine whether it intersects with the obstacle boundary. Check the continuity of the eigenvalues ​​on the line. Determine the reachable candidate points that do not intersect with obstacles and whose eigenvalues ​​change continuously as reachable points. Establish a reachability matrix based on the reachable point distribution of each grid point. Calculating the ratio of the number of reachable points of each grid point in the reachability matrix to the total number of adjacent points to obtain the reachability, performing hierarchical weighting on the reachability according to the level of the gradient descent domain in which the grid point is located to obtain a weight coefficient, combining the weight coefficient with the distance between grid points, the heading change, and the characteristic field change rate to obtain a state transition cost, and constructing a state transition cost matrix; The reachability matrix and the state transition cost matrix are combined to construct a track search space.

5. The method according to claim 1, wherein Generate a set of candidate tracks that meet the motion constraints in the track search space, and determine the optimal planned track by calculating the cumulative cost of each track in the candidate track set. In the track search space, the weight coefficients of the grid points are normalized to obtain the basic transition probability, the state transition probability distribution is calculated based on the basic transition probability and the main direction of the gradient, and the initial sampling node set is generated in the gradient descent domain at each level; The node search radius is determined according to the level of the gradient descent domain where the sampling node is located. A neighbor search is performed within the search radius to obtain a set of connectable nodes. Based on the reachability matrix, whether the connectable nodes meet the motion constraints is determined. Nodes that meet the motion constraints are connected and assigned state transition costs to construct a candidate track search tree. Performing motion constraint verification on the paths in the candidate track search tree, extracting path nodes that satisfy steering angle constraints and steering angular acceleration constraints as a control point sequence, and performing trajectory smoothing on the control point sequence to generate a candidate track set; The hierarchical cost weights are set based on the state transition cost and the gradient descent domain level, the cumulative cost of each track in the candidate track set is calculated, the track with the smallest cumulative cost is selected as the current optimal track, local sampling is performed with the control point of the optimal track as the center to obtain a new control point sequence, the new control point sequence is trajectory smoothed to generate new candidate tracks, the cumulative cost of the new candidate tracks is calculated and the optimal track is updated, and local optimization is iteratively performed until the cumulative cost converges to obtain the optimal planned track.

6. The method according to claim 1, characterized in that The flight segments are divided into multiple flight segments according to the curvature change of the optimal planned trajectory, and the flight segments are optimized online to obtain a continuous attitude sequence including: Obtaining curvature change values ​​of adjacent trajectory points on the optimal planned trajectory, determining trajectory points with a curvature change greater than a preset curvature threshold as curvature mutation points, and dividing the optimal planned trajectory into multiple flight segments based on the curvature mutation points; A sliding time window of fixed length is established for each flight segment, with the initial position of the sliding time window set at the beginning of the flight segment. Within the sliding time window, a desired G-command range and a desired roll rate range of the control point are determined based on the curvature gradient sequence at the current position, thereby generating multiple sets of candidate control input sequences. Selecting a control input sequence that satisfies flight dynamics constraints and has minimal curvature change from multiple sets of candidate control input sequences as the optimal control input, and generating a continuous attitude sequence in the current time window based on the optimal control input; The end state of the continuous attitude sequence is used as the initial state of the next sliding time window. The sliding time window is slid forward and the optimization process is repeated until the trajectory optimization of the current flight segment to be optimized is completed. After the trajectory optimization is completed for all flight segments, the optimized continuous attitude sequences of each flight segment are combined to obtain the final continuous attitude sequence.

7. The method according to claim 1, characterized in that Based on the continuous attitude sequence combined with environmental disturbance compensation, an adaptive thrust distribution strategy and a real-time attitude control strategy are generated and combined into segmented execution instructions. According to the segmented execution instructions, the trajectory tracking of the UAV to complete the flight segment includes: Calculate the desired attitude command and desired trajectory information based on the continuous attitude sequence, collect the current flight state and environmental disturbance information of the UAV, and compare the current flight state with the desired attitude command to obtain an attitude deviation value; The thrust compensation is calculated based on the attitude deviation value, and the thrust compensation is adaptively adjusted based on the environmental disturbance information to generate an adaptive thrust distribution strategy. In addition, a real-time attitude control strategy is generated based on the desired attitude command and the attitude deviation value. Dividing the flight segment into a plurality of continuous control intervals according to the characteristic points of the desired trajectory information, and combining the adaptive thrust distribution strategy and the real-time attitude control strategy to form a segmented execution instruction; Corresponding control parameters are configured in each control interval, and a smooth transition function is used at the boundary points of adjacent control intervals to achieve continuous switching of the segmented execution instructions. The UAV is controlled according to the switched segmented execution instructions to complete the trajectory tracking of the flight segment.

8. A deep learning-based autonomous obstacle avoidance and path planning system for unmanned aerial vehicles, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to obtain the drone's location information and environmental perception data; The second unit is used to construct a spatiotemporal feature matrix for environmental perception data, extract the target motion feature vector and background feature vector from the spatiotemporal feature matrix, calculate the motion trajectory projection based on the target motion feature vector, construct the environmental constraint boundary based on the background feature vector, and map the motion trajectory projection and environmental constraint boundary to the three-dimensional grid space to obtain the target-environment fusion feature field; The third unit is used to calculate the grid point reachability matrix based on the target-environment fusion feature field, calculate the state transition cost between adjacent grid points to generate a cost matrix, combine the reachability matrix and the cost matrix to construct a track search space, generate a set of candidate tracks that meet the motion constraints in the track search space, and determine the optimal planned track by calculating the cumulative cost of each track in the candidate track set; The fourth unit is used to divide the flight segments into multiple flight segments according to the curvature changes of the optimal planned trajectory, perform online trajectory optimization on the flight segments to obtain a continuous attitude sequence, generate an adaptive thrust allocation strategy and a real-time attitude control strategy based on the continuous attitude sequence combined with environmental disturbance compensation, and combine them into segmented execution instructions. According to the segmented execution instructions, the UAV is controlled to complete the trajectory tracking of the flight segment.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images