A method for unmanned aerial vehicle path planning based on semantic context-aware reinforcement learning

Abstract

The present application relates to the technical field of unmanned aerial vehicle three-dimensional path planning, and particularly relates to an unmanned aerial vehicle path planning method based on semantic context perception reinforcement learning. The present application comprises: fusing multi-source sensing data collected by the unmanned aerial vehicle to construct a multi-attribute three-dimensional map; constructing an envelope region between the current position and the target position of the unmanned aerial vehicle, and obtaining an environmental context vector; inputting the environmental context vector into a pre-trained reinforcement learning strategy network to output a set of path planning preference weights; obtaining path planning weights through path connectivity testing; constructing a comprehensive cost function based on the path planning weights; using a global path planning algorithm to search for a feasible path in the multi-attribute three-dimensional map according to the comprehensive cost function, and converting the feasible path into continuous flight instructions. The present application improves the environmental perception capability and navigation decision concentration, takes into account adaptability and planning safety, and has strong generalization capability and rapid migration characteristics.

Description

Technical Field

[0001] This invention relates to the field of UAV 3D path planning technology, specifically to a UAV path planning method based on semantic context-aware reinforcement learning. Background Technology

[0002] Traditional visual navigation for drones primarily utilizes depth cameras or parallax methods to acquire geometric data of objects, such as shape or distance, to construct a 3D environment model. This allows drones to determine the presence of obstacles ahead based on geometric data. However, drones typically cannot identify the target using raw vision. Many existing DRL (Deep Reinforcement Learning) methods require prior knowledge of the target's relative attitude and precise coordinates for navigation. If the target's location is unknown, traditional visual navigation algorithms often struggle to execute effectively.

[0003] To address this issue and optimize the performance of UAV 3D path planning, a semantic-driven navigation approach has emerged in the UAV field. This approach no longer relies solely on raw pixel data but instead uses a Reality Reconstruction Network to transform perceived information into semantically meaningful descriptions. This method establishes a unified feature space by distinguishing between navigation targets and obstacles. By reconstructing complex raw visual images into simple semantic labels, it bridges the gap between the simulated environment and the real world. Furthermore, the semantic-driven approach extracts key semantic features and removes background noise such as ground color and lighting variations, allowing the UAV to focus more on navigation decisions.

[0004] However, in semantic-driven path navigation, semantics are primarily used for end-to-end decision-making or behavioral output, lacking a structured interface for how semantics relates to planning costs and topological constraints. This makes it difficult to form interpretable, improved planning paths. Furthermore, existing UAV systems using semantic segmentation can only distinguish geometric objects, failing to differentiate between contextual situations such as narrow passages, high-risk areas, and areas that can be soft-crossed. This results in a consistent planning style in complex environments, lacking environmental awareness. Additionally, traditional planners use fixed cost weights, leading to poor adaptability to environments with varying structures. While end-to-end reinforcement learning can adapt, it directly controls flight, making it difficult to guarantee path feasibility and safety boundaries. Summary of the Invention

[0005] The purpose of this invention is to solve the problems existing in the prior art and to propose a UAV path planning method based on semantic context-aware reinforcement learning.

[0006] To achieve the above objectives, the technical solution provided by this invention is as follows:

[0007] A method for UAV path planning based on semantic context-aware reinforcement learning, comprising:

[0008] Multi-source sensor data collected by drones are fused to construct a multi-attribute 3D map that includes spatial occupancy information, distance information, and semantic risk attributes.

[0009] A three-dimensional cube is constructed using the current position of the UAV and the target position as the diagonal vertices as the envelope region, and the environmental context vector is obtained based on the envelope region;

[0010] The environmental context vector is input into a pre-trained reinforcement learning policy network, which outputs a set of path planning preference weights.

[0011] A temporary cost function is constructed based on the path planning preference weights, and path connectivity is tested in a multi-attribute 3D map. If the connectivity test is passed, the current path planning preference weights are used as the path planning weights; if the connectivity test is not passed, the preset conservative weights are used as the path planning weights. A comprehensive cost function is constructed based on the path planning weights.

[0012] Based on the environment in which the UAV is located, a corresponding global path planning algorithm is selected. Using the selected global path planning algorithm, a feasible path is searched in a multi-attribute 3D map according to the comprehensive cost function, and the feasible path is converted into a continuous flight command.

[0013] Furthermore, the process of fusing multi-source sensor data collected by the UAV to construct a multi-attribute 3D map including spatial occupancy information, distance information, and semantic risk attributes includes:

[0014] Using drones to collect sensor data, the sensor data is then subjected to lidar motion distortion correction, voxel grid filtering, and camera distortion correction.

[0015] An occupied grid map is constructed based on the processed sensor data. The occupied grid map contains occupied voxels and idle voxels. Occupied voxels are voxels with obstacles, and idle voxels are voxels without obstacles.

[0016] Calculate the Euclidean distance from the empty voxel to the nearest obstacle and the repulsion gradient pointing outward from the obstacle based on the occupied grid map, and construct a three-dimensional composite map containing spatial occupancy information and distance information;

[0017] The images collected by the drone are semantically parsed to identify semantic categories. Based on the preset semantic information corresponding to the risk table, different semantic categories are mapped to corresponding semantic risk weights to generate a two-dimensional risk image.

[0018] By acquiring the real-time pose information of the drone and the intrinsic parameter matrix of the camera, the two-dimensional risk image is spatially mapped and fused with the three-dimensional composite map to form a multi-attribute three-dimensional map.

[0019] Furthermore, obtaining the environmental context vector based on the envelope region includes:

[0020] Semantic information of all voxels within the envelope region is extracted to obtain a subset of the dataset;

[0021] Based on the proportion of occupied voxels to free voxels in the subset dataset, the ratio of obstacle density to passable space is obtained.

[0022] Calculate the width of the narrowest passage within the envelope region based on the distance of each free voxel to the nearest obstacle;

[0023] By statistically analyzing the volume percentage of voxels occupying the voxel and the entire envelope region, information on the volume percentage of high-risk voxels can be obtained.

[0024] The semantic risk mean of the envelope region is calculated based on the semantic risk weight of each voxel within the envelope region;

[0025] The environmental context vector is obtained based on the width of the narrowest channel, the proportion of high-risk voxel volume, the mean semantic risk, the current linear velocity of the drone, and the angle between the current drone nose orientation and the target point.

[0026] Furthermore, it also includes applying safety constraints to the path planning preference weights before constructing the temporary cost function:

[0027] Check whether the path planning preference weights fall within the preset safety parameter space;

[0028] If the path planning preference weights exceed the safety parameter space, they are modified to the safety boundary of the safety parameter space through pruning or spatial projection techniques.

[0029] Furthermore, the comprehensive cost function is expressed by the formula:

[0030]

[0031] in, For the first The overall cost of an individual element, where k is the number of path planning weights. For the first Each path planning weight, For the first The first individual element Sub-costs corresponding to each path planning weight.

[0032] Furthermore, the step of selecting the corresponding global path planning algorithm based on the environment includes:

[0033] If the environment of the envelope region is indoors, then the A* algorithm is selected as the global path planning algorithm;

[0034] If the envelope region is located outdoors, then the RRT* algorithm is selected as the global path planning algorithm.

[0035] Furthermore, the path planning preference weights include: path length preference weight, obstacle gap preference weight, risk aversion preference weight, energy consumption preference weight, and path smoothness preference weight.

[0036] Furthermore, this also includes smoothing the feasible paths after they are obtained:

[0037] The B-spline algorithm is used to transform the polyline path into a curved path by using the third derivative of the path trajectory as a smoothness metric.

[0038] Convert the curve trajectory into coordinate points with time information;

[0039] The coordinate points are overlaid onto a multi-attribute 3D map for evaluation. If a collision is detected, the current coordinate point is moved along the bisector of the angle between the vectors pointing to the two adjacent nodes until the collision is eliminated.

[0040] Furthermore, the reinforcement learning policy network utilizes the Berlin noise algorithm to generate virtual simulation environments with increasing difficulty.

[0041] Compared with the prior art, the significant advantages of this invention are:

[0042] (1) Enhancing environmental perception and navigation decision focus: By constructing a three-dimensional composite map containing spatial occupancy and distance information, and integrating semantic risk weights to form semantically enhanced environmental labels, the gap between the simulated environment and the real world is effectively narrowed. The environmental features extracted from the envelope region are combined into an environmental context vector, which can accurately describe the path planning mode that should be selected in the current environment.

[0043] (2) Balancing Adaptability and Planning Safety: The reinforcement learning policy network does not directly output flight action control sequences, but only outputs parameter weights used to adjust the path planning cost function. Through safety constraints, the parameters are always kept within the safe parameter space, and when a path cannot be generated, a backoff mechanism is used to restore the preset conservative weights, thereby ensuring the feasibility, safety, and interpretability of the path planning process.

[0044] (3) It has strong generalization ability and fast transfer characteristics: It adopts a course-based environment generation method from simple to complex, so that the reinforcement learning strategy can grasp the mapping relationship between the environmental context and the planning preference. By introducing a lightweight parameter adaptation module into the output layer, the policy network can complete the transfer by updating only a few parameters when facing a new environmental structure. It has good stability and generalization ability under the condition of no topology and perceived noise disturbance. Attached Figure Description

[0045] Figure 1 This is a flowchart of a UAV path planning method based on semantic context-aware reinforcement learning according to the present invention.

[0046] Figure 2 This is a flowchart illustrating the fusion of semantic risk and 3D map to form a 3D composite map according to the present invention.

[0047] Figure 3 This is a schematic diagram illustrating how the environmental context is processed by an actor-critic network to obtain preference weights in this invention.

[0048] Figure 4 This is a schematic diagram of the UAV following a path after the reinforcement learning phase ends in this invention.

[0049] Figure 5 This is a comparison diagram between the present invention, which inserts a residual bottleneck adapter into the output layer of the actor-critic strategy network, and the prior art. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0051] This invention provides a method for UAV path planning based on semantic context-aware reinforcement learning, such as... Figure 1 As shown, it includes the following steps:

[0052] (1) Constructing a basic map: Using the cameras, lidar, and other sensors mounted on the UAV to perceive the outside world, the surrounding environment is continuously scanned during the flight of the UAV to obtain sensor data (including RGB images acquired by the camera and information from the position sensor). The RGB images acquired by the camera and the information from the position sensor are synchronized according to the time nodes, and the sensor data is corrected for lidar motion distortion, voxel grid filtering, and camera distortion using existing methods. At the same time, a three-dimensional map is constructed using the existing Bayesian algorithm, and an occupied grid map is drawn for each voxel to determine whether there are obstacles. The occupied grid map contains occupied voxels and idle voxels. Occupied voxels are voxels with obstacles, and idle voxels are voxels without obstacles. The Euclidean distance from the idle voxel to the nearest obstacle and the repulsion gradient pointing outward from the obstacle are calculated based on the occupied grid map. Finally, a three-dimensional composite map containing spatial occupancy information and distance information (distance gradient information) is output for use in subsequent steps.

[0053] (2) Establish a semantic risk map and construct an environmental context: such as Figure 2 As shown, semantic analysis and risk mapping are performed on RGB images based on a 3D composite map to obtain a semantically enhanced environment that integrates semantics and map. The envelope region is then planned and combined into a fixed-dimensional environment context vector based on the semantically enhanced environment. This process includes:

[0054] (2-1) Perform visual semantic segmentation and mapping. Receive the original RGB image stream from the camera, use a traditional deep learning model to determine the type of objects in the RGB image, and generate a semantic segmentation layer with the same size as the original image, using different colors to represent different categories of objects.

[0055] After identifying objects, a semantic information-to-risk table is predefined based on physical experience or prior knowledge. This table converts object types into risk levels, providing quantitative data for subsequent obstacle avoidance, as shown in Table 1.

[0056] Table 1. Risk Table Corresponding to Semantic Information

[0057] Category catalogue Semantic risk weight Sky 0 Grass 0.1 Crown 0.5 Wire 1.0 Building 1.5

[0058] The table assigns corresponding semantic risk weights based on different semantic categories, and uses the type labels in the previously obtained semantic segmentation layer to match the risk table corresponding to the semantic information to regenerate a two-dimensional risk image.

[0059] (2-2) Semantic projection and multi-level map fusion: utilizing the camera's intrinsic parameter matrix By establishing a mapping relationship between the pixel coordinates of the 2D image and 3D space based on the position and attitude information of the UAV, and then using the ray projection method, the mapping relationship between the 2D risk image and 3D space is found. The semantic risk weight of each pixel is assigned to the corresponding voxel in the 3D space, forming a preliminary 3D semantic risk layer. This process can be described by the following formula:

[0060]

[0061]

[0062] in, This is the intrinsic parameter matrix of the camera. Focal length, expressed in pixels, is a value preset by the camera at the factory. Let be the coordinates of the camera's optical center on the image plane, and s be the tilt factor, which is set to 0 on modern cameras. These are the actual coordinates of the object on a real-world map. These represent the camera's orientation and geometric position relative to the drone's IMU, respectively. Let be the estimated rotation and translation states of the UAV at time t, respectively; d be the distance from the UAV's camera to the obstacle; and u and v be the horizontal and vertical positions of a point in the image. The camera spatial point in the formula ( ) is the position of a pixel in an image corresponding to a certain object in three-dimensional space, with the optical center of the camera as the origin, and the rotation combination part ( The camera's deflection angle relative to the real world is calculated by superimposing the camera's installation angle relative to the drone and the drone's own motion angle. The total translation component ( The position of the camera relative to the real world is obtained by adding the position of the object observed by the camera to the position measured by the drone's IMU in the real world.

[0063] The above steps ensure that the 3D composite map and the semantic risk layer are consistent in coordinates, and finally output a multi-attribute 3D map that includes whether each voxel is an obstacle, the distance of each free voxel from the obstacle, and the semantic risk weight.

[0064] (2-3) Select the current position of the drone as the starting point and the end point as the target position, and construct a three-dimensional cube with the starting point and the end point as the diagonal vertices, that is, construct the envelope region to eliminate interference from irrelevant environments behind and in the distance.

[0065] An envelope region is obtained by calculating the maxima and minima of the X, Y, Z 3D coordinates of the envelope region in the world coordinate system using the existing axis-aligned bounding box technique (AABB). This process is represented as follows:

[0066]

[0067] Where AABB represents the constructed envelope region. , Let x be the minimum and maximum x-coordinates of a cube with the starting and ending points as diagonal vertices, respectively. Similarly... , These are the minimum and maximum values ​​of the y-coordinate, respectively. , These are the minimum and maximum values ​​of the z-coordinate, respectively. This will filter out most of the map data outside the envelope area to reduce data complexity.

[0068] (2-4) The spatial indexing algorithm is invoked to identify and extract all voxels and semantic information within the envelope region constructed in (2-3) to obtain a subset containing local location and semantic risk weights. Based on the proportion of occupied voxels to idle voxels in the subset, the ratio of obstacle density to passable space is determined. The width of the narrowest passage within the envelope region is calculated based on the distance of each idle voxel to the nearest obstacle. Simultaneously, the proportion of high-risk voxels representing obstacles to the total voxel volume of the entire envelope region is statistically analyzed.

[0069]

[0070] in, This indicates the percentage of high-risk voxels by volume. This represents the number of voxels with a semantic risk weight > 0.8. This indicates the number of voxels in the entire envelope region.

[0071] (2-5) Based on the width of the narrowest channel and the proportion of high-risk voxels obtained in (2-4), normalize them to the [0,1] interval, and concatenate these values ​​into a feature vector according to a preset fixed order. For example, concatenate them into... ,in, Represents the environment context vector. The width of the narrowest channel after normalization. The percentage of high-risk voxels obtained in (2-4) The mean of semantic risk. , This indicates the number of voxels within the envelope region. Indicates the first term within the envelope region Semantic risk weights of individual elements The current linear velocity of the drone, This is the angle between the drone's current nose-up orientation and the target point.

[0072] (3) Generate path planning function: Input the environment context vector obtained in step (2) into the pre-trained reinforcement learning policy network (in this embodiment, the reinforcement learning policy network is the actor-critic policy network). The network will output parameters for adjusting the path planning cost according to various parameters, and finally generate the path planning function.

[0073] Considering that traditional reinforcement learning path planning algorithms directly output control sequences with a very large vector dimension space, which places high demands on the computing power of the UAV's onboard computer, traditional methods struggle to find a good balance between performance and computing power. Therefore, this embodiment chooses to output parameters used to adjust the path planning cost function. The flight control chip, independent of the onboard computer, obtains these parameters and performs its own calculations to generate the control sequence. This method effectively reduces the dimensionality of the output parameters, avoiding the conflict between computing power and performance caused by dimensionality explosion. The process is as follows:

[0074] (3-1) For example Figure 3 As shown, the environmental context vector obtained in (2-5) is used as input, and after passing through existing multi-layer nonlinear transformation techniques, the output is a set of path planning preference weights. , , , ],in, Indicates path length preference weights. Indicates the obstacle gap preference weight, This indicates the weights of risk aversion and energy consumption preference. This represents the path smoothness preference weight. It checks whether the path planning preference weight falls within the preset safety parameter space. If the path planning preference weight is found to exceed the preset safety parameter space during testing—for example, by compressing obstacle avoidance gap weights in an excessive pursuit of speed—then the path planning preference weight is modified to within the safety boundary of the safety parameter space using pruning or spatial projection techniques, and a temporary cost function is constructed.

[0075]

[0076] in, The first output directly from the actor-critic strategy network Each path planning preference weight For the first The first individual element The sub-costs corresponding to each path planning preference weight, where k is the number of path planning preference weights. In this embodiment, k is 4, meaning that path evaluation is performed based on four path planning preference weights.

[0077] (3-2) After constructing the temporary cost function in the previous step, a breadth-first search algorithm is used to perform a preliminary path connectivity test on the multi-attribute 3D map to determine whether the currently generated path planning preference weights have physical paths that satisfy the flight constraints. If there are physical paths that satisfy the flight constraints, the current path planning preference weights are used as path planning weights and sent to the subsequent path planners for subsequent steps. If there are no physical paths that satisfy the flight constraints, a backoff mechanism is initiated, and the preset conservative weights are directly used as path planning weights to plan subsequent flights.

[0078] (4) Determine the path planning: Construct a comprehensive cost function. Based on the comprehensive cost function, output the cumulative cost through the value function and provide it to the global path planning algorithm. After the global path planning algorithm calculates the feasible path, it is handed over to the control terminal for use. This process includes the following tasks:

[0079] (4-1) In step (3-2), the path planning weights verified by the preliminary connectivity test are mapped to the coefficients in the cost function. The comprehensive cost under the combined influence of the path planning weights and cost terms is calculated by weighted summation, and the comprehensive cost function is constructed accordingly:

[0080]

[0081] in, For the first The overall cost of an individual element, where k is the number of path planning weights, equivalent to the number of path planning preference weights. The output of (3-2) after passing through the actor-critic strategy network and undergoing security constraints is the first... Each path planning weight, For the first The first individual element Each path planning weight corresponds to a sub-cost, which includes path length cost, obstacle gap cost, semantic risk cost, and smoothness cost. These costs correspond to each dimension of the vector output by the neural network.

[0082] (4-2) After obtaining the comprehensive cost, different global path planning algorithms are selected for specific flight control based on the environment. For example, the A* algorithm is used for path planning in indoor spaces:

[0083]

[0084] in, From the starting point to the current node (current voxel) The cumulative cost, that is, the cost incurred from the starting point to the current node, is the path accumulation of the cost function in (4-1). To select the current node The cost of the next target point This is the weighted Euclidean distance from the current node to the destination. The weighted Euclidean distance is expressed as follows:

[0085]

[0086] in, This represents the distance between the current point and the destination. Indicates the current node, for The spatial coordinates of the node Indicates the end point. for The spatial coordinates of the node The comprehensive cost function obtained from (4-1) is given.

[0087] In outdoor spaces, the RTT (Rapidly-exploring Random Tree Star) algorithm can be used to detect connectivity. This process avoids high-risk semantic regions and tends to traverse safe areas with larger distance values.

[0088] (4-3) In the path planning process, the drone is assumed to be a point, but the actual drone can be approximated as a sphere with a certain volume. Therefore, it is necessary to check whether there is a collision risk between the searched path points due to the size of the drone:

[0089] Set the safe collision radius for the drone (e.g., take the drone's rotor radius). When the path planning algorithm attempts to select or expand a new candidate node, it directly calls the pre-stored distance gradient information of the coordinate point corresponding to the candidate node in the multi-attribute 3D map. It extracts the Euclidean distance from the candidate node to the nearest obstacle object from the map data. This Euclidean distance is then compared with a preset safe collision radius. Compare:

[0090] If the Euclidean distance from the candidate node to the nearest obstacle object is less than If the candidate node is located within the physical collision boundary caused by the size of the drone, it is determined that there is a risk of collision.

[0091] If the Euclidean distance from the candidate node to the nearest obstacle object is greater than or equal to If the candidate node is found to be safe in physical space, then it is determined that the candidate node is safe.

[0092] For candidate nodes that are determined to have a collision risk, they are directly marked as unreachable and removed from the current search queue or path tree branch.

[0093] (4-4) The flight process of UAVs is extremely sensitive to the smoothness of the path. If there are sharp turns in the generated path, it may affect the hardware structure of the aircraft. Therefore, the broken lines in the planned path should be transformed into curves. Based on the candidate nodes that are safe in physical space searched in (4-3), the trajectory optimization technology is used to smooth the path. While ensuring safety, the changes in control commands should be minimized to save energy.

[0094] In physics, the first derivative with respect to position represents velocity, the second derivative represents acceleration, and the third derivative represents smoothness. Calculate the smoothness of the current trajectory:

[0095]

[0096] in, This indicates the smoothness of the entire trajectory; the smaller the value, the smoother the trajectory. For drones The three-dimensional coordinates at time [time]. This represents the total flight time of the planned path (trajectory). Taking 3 represents the third derivative, which indicates the smoothness.

[0097] The smoothing algorithm is implemented using the existing B-Splines algorithm. The mathematical expression for the B-Splines curve is:

[0098]

[0099] in, Indicates a smoothed continuous path. Represents the first result obtained by the A* or RRT* algorithm. The coordinates of the control points Indicates the first control points A B-spline basis function of order B is a continuous function. The value is usually 3, representing three B-splines calculations, obtained through recursion:

[0100]

[0101]

[0102] in, It is an independent variable, representing the values ​​taken at equal intervals between [0,1]. Each value can be abstractly understood as a curve that can be decomposed into... A straight line segment, and then in You can pass it at any time Obtain the acceleration and velocity at that moment. This is the index of the control point, with a value ranging from [0, A * total number of points planned by the algorithm - 1]. Indicates the first basis functions Scope of application Indicates adjacent The next basis function, express The range of influence that the first-order basis functions can have, and so on, as... Gradually approaching Afterward, the value of the fraction will gradually approach 0, thus achieving a smoothing effect.

[0103] (4-5) The smooth trajectory obtained after step (4-4) is transformed into a series of coordinate points with time information. These coordinate points are then overlaid on a multi-attribute 3D map for final evaluation, ensuring that the coordinate points are entirely within the feasible area. If a collision is still detected, the collision is then determined based on the node where the collision occurred. Previous node With the next node Constructing vectors , ,use This will allow you to obtain the direction of the obstacle. , The angle bisector of the included angle is used to move the point of collision by one voxel unit along this direction, and the detection is repeated until there is no collision.

[0104] (5) Path planning execution: This step involves actual navigation after obtaining the path, such as... Figure 4 As shown, ROS (Robot Operating System) is a commonly used task publishing operating system in the field of drones and robots. It can publish the non-collision coordinates obtained in step (4) to the drone flight controller. The PX4 flight control system running in the drone flight controller can then automatically go to the target point according to the command issued by ROS, thereby realizing the function of drone following the planned path for flight.

[0105] (6) Cross-scene generalization and policy stability: The complexity of the environment is gradually increased during the training process, so that the actor-critic policy network can better learn the mapping relationship between the environmental context and planning preferences, rather than just memorizing map details.

[0106] (6-1) To adopt a curriculum-based learning approach, the first step is to construct a curriculum-based environment. This involves defining evaluation metrics for environmental complexity (obstacle occupancy rate, passage width, etc.) and using Procedural Content Generation (PCG) to generate scenarios that gradually increase from the simplest straight-line flight scenarios to more complex scenarios with multiple obstacles and high-risk areas. The specific steps in this process are as follows:

[0107] The existing Berlin noise algorithm is used to generate undulating terrain and irregular obstacle groups. The code is as follows:

[0108] def generate_perlin_heightmap(width, height, scale, octaves, persistence, lacunarity): # The larger the octaves, the rougher the generated terrain surface, and the more realistic the simulation of terrain.

[0109] world = np.zeros((width, height))

[0110] for i in range(width):

[0111] for j in range(height):

[0112] # i / scale determines the frequency; the larger the scale, the flatter the terrain.

[0113] world[i][j] = pnoise2(i / scale, j / scale, octaves=octaves,

[0114] persistence = persistence,

[0115] lacunarity = lacunarity,

[0116] repeatx=1024,

[0117] repeaty=1024,

[0118] base=0).

[0119] After processing with the Berlin noise algorithm, a two-dimensional height map H(x, y) is obtained. For a voxel (x, y, z) in the coordinate space, if z ≤ H(x, y), the voxel is marked as an obstacle; for example, z ≤ 0.2 is marked as grass, and z > 0.8 is marked as rock. Here, x represents the x-axis coordinate of the current voxel, y represents the y-axis coordinate of the current voxel, and z represents the z-axis coordinate of the current voxel. The generated terrain is saved as a grayscale image in .png format, with the height mapped from low to high to grayscale values ​​in the range [0, 255]. The grayscale image is imported into the Gazebo physics simulation environment, where Gazebo automatically generates 3D terrain with collision volumes for subsequent training.

[0120] (6-2) In a course-based environment, the actor-critic strategy network is trained with increasing difficulty. The parameters of the Berlin noise algorithm are adjusted to generate virtual simulation environments with increasing difficulty. The generated terrain gradually changes from simple flat land to complex mountains. Multiple training sessions are conducted in the increasingly complex terrain. This process adopts the idea of ​​pre-training. As the course difficulty increases, the network will gradually learn how to maintain security by adjusting weights under various extreme topologies.

[0121] (6-3) Insert a lightweight parameter adaptation module (e.g., residual bottleneck adapter) into the output layer of the actor-critic strategy network. After the basic training is completed, the newly inserted parameter block can be used to accelerate the tracking speed in new scenarios in the future.

[0122] This process is like Figure 5 As shown, assuming the output layer of the last layer of the actor-critic strategy network is... The second to last layer is The final output Then the superimposed output Passed to the last output layer ,in, express The output of the layer, The output of the residual bottleneck adapter is represented as follows:

[0123]

[0124] in, Represents a dimension reduction matrix. Represents an upgraded matrix. Represents a non-linear activation function. and This represents the bias vector.

[0125] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for UAV path planning based on semantic context-aware reinforcement learning, characterized in that, include: Multi-source sensor data collected by drones are fused to construct a multi-attribute 3D map that includes spatial occupancy information, distance information, and semantic risk attributes. A three-dimensional cube is constructed using the current position of the UAV and the target position as the diagonal vertices as the envelope region, and the environmental context vector is obtained based on the envelope region; The environmental context vector is input into a pre-trained reinforcement learning policy network, which outputs a set of path planning preference weights. A temporary cost function is constructed based on path planning preference weights, and path connectivity is tested in a multi-attribute 3D map. If the connectivity test is passed, the current path planning preference weight will be used as the path planning weight; if the connectivity test is not passed, the preset conservative weight will be used as the path planning weight. Construct a comprehensive cost function based on path planning weights; Based on the environment in which the UAV is located, a corresponding global path planning algorithm is selected. Using the selected global path planning algorithm, a feasible path is searched in a multi-attribute 3D map according to the comprehensive cost function, and the feasible path is converted into a continuous flight command.

2. The UAV path planning method based on semantic context-aware reinforcement learning according to claim 1, characterized in that, The process of fusing multi-source sensor data collected by the UAV to construct a multi-attribute 3D map that includes spatial occupancy information, distance information, and semantic risk attributes includes: Using drones to collect sensor data, the sensor data is then subjected to lidar motion distortion correction, voxel grid filtering, and camera distortion correction. An occupied grid map is constructed based on the processed sensor data. The occupied grid map contains occupied voxels and idle voxels. Occupied voxels are voxels with obstacles, and idle voxels are voxels without obstacles. Calculate the Euclidean distance from the empty voxel to the nearest obstacle and the repulsion gradient pointing outward from the obstacle based on the occupied grid map, and construct a three-dimensional composite map containing spatial occupancy information and distance information; The images collected by the drone are semantically parsed to identify semantic categories. Based on the preset semantic information corresponding to the risk table, different semantic categories are mapped to corresponding semantic risk weights to generate a two-dimensional risk image. By acquiring the real-time pose information of the drone and the intrinsic parameter matrix of the camera, the two-dimensional risk image is spatially mapped and fused with the three-dimensional composite map to form a multi-attribute three-dimensional map.

3. The UAV path planning method based on semantic context-aware reinforcement learning according to claim 1, characterized in that, The process of obtaining the environmental context vector based on the envelope region includes: Semantic information of all voxels within the envelope region is extracted to obtain a subset of the dataset; Based on the proportion of occupied voxels to free voxels in the subset dataset, the ratio of obstacle density to passable space is obtained. Calculate the width of the narrowest passage within the envelope region based on the distance of each free voxel to the nearest obstacle; By statistically analyzing the volume percentage of voxels occupying the voxel and the entire envelope region, information on the volume percentage of high-risk voxels can be obtained. The semantic risk mean of the envelope region is calculated based on the semantic risk weight of each voxel within the envelope region; The environmental context vector is obtained based on the width of the narrowest channel, the proportion of high-risk voxel volume, the mean semantic risk, the current linear velocity of the drone, and the angle between the current drone nose orientation and the target point.

4. The UAV path planning method based on semantic context-aware reinforcement learning according to claim 1, characterized in that, This also includes applying safety constraints to the path planning preference weights before constructing the temporary cost function: Check whether the path planning preference weights fall within the preset safety parameter space; If the path planning preference weights exceed the safety parameter space, they are modified to the safety boundary of the safety parameter space through pruning or spatial projection techniques.

5. The UAV path planning method based on semantic context-aware reinforcement learning according to claim 1, characterized in that, The comprehensive cost function is expressed by the following formula: in, For the first The overall cost of an individual element, where k is the number of path planning weights. For the first Each path planning weight, For the first The first individual element Sub-costs corresponding to each path planning weight.

6. The UAV path planning method based on semantic context-aware reinforcement learning according to claim 1, characterized in that, The global path planning algorithm based on the environment includes: If the environment of the envelope region is indoors, then the A* algorithm is selected as the global path planning algorithm; If the envelope region is located outdoors, then the RRT* algorithm is selected as the global path planning algorithm.

7. The UAV path planning method based on semantic context-aware reinforcement learning according to claim 1, characterized in that, The path planning preference weights include: path length preference weight, obstacle gap preference weight, risk aversion preference weight, energy consumption preference weight, and path smoothness preference weight.

8. The UAV path planning method based on semantic context-aware reinforcement learning according to claim 1, characterized in that, This also includes smoothing the feasible paths after they are obtained: The B-spline algorithm is used to transform the polyline path into a curved path by using the third derivative of the path trajectory as a smoothness metric. Convert the curve trajectory into coordinate points with time information; The coordinate points are overlaid onto a multi-attribute 3D map for evaluation. If a collision is detected, the current coordinate point is moved along the bisector of the angle between the vectors pointing to the two adjacent nodes until the collision is eliminated.

9. The UAV path planning method based on semantic context-aware reinforcement learning according to claim 1, characterized in that, The reinforcement learning policy network uses the Berlin noise algorithm to generate virtual simulation environments with increasing difficulty.

Images