Fire-fighting unmanned aerial vehicle autonomous obstacle avoidance flight control method based on visual measurement

Abstract

The application discloses a fire-fighting unmanned aerial vehicle autonomous obstacle avoidance flight control method based on visual measurement, relates to the technical field of fire-fighting unmanned aerial vehicle flight control, and comprises the following steps: acquiring a continuous original image frame sequence collected by an on-board visual sensor of an unmanned aerial vehicle, identifying and segmenting dynamic and static obstacle regions through pixel-level visual analysis, and outputting an obstacle region segmentation graph; synchronously solving instant position data of the unmanned aerial vehicle and aligning the instant position data with the segmentation graph in space and time to generate a real-time environment threat field of the unmanned aerial vehicle body coordinate system; constructing a synthetic virtual potential field containing a repulsive potential field and a guide potential field, calculating a theoretical flight speed vector by using a gradient descent method, generating executable flight control instructions through smoothness constraint and dynamic school test, and driving the unmanned aerial vehicle to adjust the attitude and position. The application can improve the obstacle avoidance sensing precision of the fire-fighting unmanned aerial vehicle, optimize the environment threat representation fitting degree, and guarantee the stable execution of autonomous obstacle avoidance flight.

Description

Technical Field

[0001] This invention belongs to the field of flight control technology for firefighting drones, specifically a visual measurement-based autonomous obstacle avoidance flight control method for firefighting drones. Background Technology

[0002] Existing autonomous obstacle avoidance flight methods for firefighting drones mostly employ conventional visual obstacle detection schemes. After acquiring environmental images through onboard visual sensors, obstacles are identified using target-level detection. This approach only completes overall obstacle determination and does not perform pixel-level visual analysis of the images. It cannot distinguish the regional features of dynamic and static obstacles, nor can it output refined obstacle region segmentation results. In traditional schemes, drone pose calculation and obstacle perception data are processed independently without spatiotemporal alignment. Environmental modeling often relies on global coordinate systems to construct models, failing to form an environmental threat representation based on the drone itself. Conventional virtual potential field construction simply superimposes obstacle and waypoint potential fields. Flight speed calculations lack smoothness constraints and dynamic feasibility verification, resulting in insufficient adaptability for flight control command execution.

[0003] Conventional visual obstacle avoidance lacks pixel-level analysis, resulting in insufficient accuracy in representing obstacle threat areas. It fails to precisely distinguish between the differentiated threats of dynamic and static obstacles, and the spatiotemporal misalignment between pose data and obstacle information leads to a mismatch between environmental data and the actual flight state of the UAV. Furthermore, the global coordinate system environmental model cannot adapt to the real-time attitude changes of the UAV. This invention aims to identify and segment dynamic and static obstacle regions through pixel-level visual analysis and output obstacle region segmentation maps. By spatiotemporally aligning the UAV's real-time pose data with the obstacle region segmentation maps, a real-time environmental threat field is generated with the UAV's body coordinate system as a reference. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art; Therefore, this invention proposes an autonomous obstacle avoidance flight control method for firefighting drones based on vision measurement, comprising: Acquire a continuous sequence of raw image frames of the flight environment captured by the UAV's onboard visual sensor; Pixel-level visual analysis is performed on the continuous raw image frame sequence to identify and segment dynamic obstacle regions and static obstacle regions with potential collision threats, and an obstacle region segmentation map is output. The instantaneous pose data of the UAV is calculated synchronously, and the instantaneous pose data is spatiotemporally aligned with the obstacle area segmentation map to generate a real-time environmental threat field with the UAV body coordinate system as a reference. Based on the real-time environmental threat field, a synthetic virtual potential field with a repulsive potential field and a guiding potential field is constructed online, wherein the repulsive potential field is generated by the obstacle region and the guiding potential field is generated by the preset flight waypoint; In the synthetic virtual potential field, the gradient descent method is used to iteratively calculate the theoretical flight velocity vector of the UAV at the next moment; The smoothness constraint and dynamic feasibility verification of the theoretical flight velocity vector at the next moment are performed to generate executable flight control commands; The executable flight control commands are sent to the UAV's flight controller to drive the UAV to complete attitude and position adjustments.

[0005] Furthermore, pixel-level visual analysis is performed on the continuous original image frame sequence to identify and segment dynamic obstacle regions and static obstacle regions with potential collision threats, including: For each frame in the continuous original image frame sequence, a dense optical flow calculation algorithm is applied to obtain the motion vector of each pixel in the image, forming an optical flow field; Based on the optical flow field, a set of pixels with a coherent motion pattern is separated by motion vector clustering and region growing techniques, and the set of pixels is labeled as a candidate region for dynamic obstacles. Edge detection and texture feature extraction are performed simultaneously on the current image frame. Combined with the prior geometric shape model, contour regions that are significantly different from the background in the image are identified, and these contour regions are marked as static obstacle candidate regions. The dynamic obstacle candidate region and the static obstacle candidate region are merged, and noise regions with an area smaller than a preset threshold are filtered out to generate an obstacle region segmentation map containing obstacle category labels and contour information.

[0006] Furthermore, the simultaneous calculation of the UAV's real-time pose data, and the spatiotemporal alignment of the real-time pose data with the obstacle region segmentation map, to generate a real-time environmental threat field with the UAV's body coordinate system as a reference, includes: Read the angular velocity and acceleration data output by the UAV's inertial measurement unit, as well as the position data output by the global navigation satellite system; The angular velocity, acceleration, and position data are fused and calculated using a sensor fusion filter to output the six-degree-of-freedom pose of the UAV at the current moment, which includes three-dimensional position and three-dimensional attitude. Based on the six-degree-of-freedom pose and the intrinsic and extrinsic parameter matrices of the airborne vision sensor, the obstacle contour points corresponding to each pixel in the obstacle region segmentation image are back-projected from the image coordinate system to the three-dimensional space under the UAV body coordinate system. In the body coordinate system, a threat value is assigned to each projected 3D obstacle point. The threat value is calculated based on the obstacle type and its relative distance from the drone. All 3D point clouds with threat values ​​together constitute the real-time environmental threat field.

[0007] Furthermore, based on the aforementioned real-time environmental threat field, a synthetic virtual potential field with a repulsive potential field and a guiding potential field is constructed online, including: In the real-time environmental threat field, a repulsion potential function is defined with each three-dimensional obstacle point as the center. The value of the repulsion potential function is inversely proportional to the distance from the three-dimensional obstacle point to the current position of the UAV and directly proportional to the threat level carried by the three-dimensional obstacle point. A guiding potential function is defined with the spatial position of the next target waypoint set in the UAV flight mission as the center. The value of the guiding potential function is proportional to the distance from the current position of the UAV to the target waypoint. On the spatial grid map of the UAV's body coordinate system, the values ​​of the repulsive potential function and the guiding potential function on each spatial grid are vector-superimposed to generate the synthetic virtual potential field. The potential gradient direction of the synthetic virtual potential field at any grid indicates the direction of the virtual resultant force that the UAV should experience at the corresponding position.

[0008] Furthermore, in the synthesized virtual potential field, the theoretical flight velocity vector of the UAV at the next moment is iteratively calculated using the gradient descent method, including: The current position of the UAV in the body coordinate system is used as the starting point for the gradient descent method iteration; Calculate the potential energy gradient of the starting point of the iteration in the synthetic virtual potential field, where the potential energy gradient is a three-dimensional vector; The opposite direction of the potential energy gradient is normalized, and the resulting direction is taken as the expected movement direction of the UAV at the current position. The desired direction of movement is multiplied by a baseline velocity scalar that is dynamically adjusted based on the distance from the UAV to the target waypoint to obtain the theoretical flight velocity vector for the next moment.

[0009] Furthermore, the theoretical flight velocity vector at the next moment is subjected to smoothness constraints and dynamic feasibility verification to generate executable flight control commands, including: Obtain the actual flight speed vector of the UAV in the previous control cycle; Calculate the difference between the theoretical flight speed vector and the actual flight speed vector at the next moment, and apply a low-pass filter to the difference to smooth the sudden changes in speed command; The smoothed velocity vector is input into the inverse dynamics model of the UAV. The inverse dynamics model calculates the thrust of each rotor required to achieve the change of the smoothed velocity vector based on the mass, moment of inertia, maximum thrust and torque parameters of the UAV. The calculated thrust of each rotor is verified to be within the range of thrust that the motor can provide. If it exceeds the range, the thrust is limited to the maximum value, and the corresponding achievable velocity vector is recalculated. The velocity vector and its corresponding attitude angle command together constitute the executable flight control command.

[0010] Furthermore, it also includes the steps of continuously monitoring the flight status and updating the potential field online after generating executable flight control commands: After executing the current control command, a new continuous sequence of raw image frames and real-time pose data of the UAV are acquired again. Based on the new data, the steps from visual analysis to generating a real-time environmental threat field are repeated, and the real-time environmental threat field is updated. In the synthesized virtual potential field, the weighting coefficients of the guiding potential function are dynamically adjusted according to the current position of the UAV and the target waypoint position, so that the repulsive potential field dominates when approaching an obstacle, and the guiding potential field dominates when in open space.

[0011] Furthermore, the dynamic adjustment of the weighting coefficients of the guiding potential function includes: Calculate the ratio of the threat level of the nearest obstacle to the drone in the current real-time environmental threat field to its distance, and use it as an indicator of environmental congestion. Define a smoothing function that is negatively correlated with the environmental congestion index, and use the output value of the smoothing function as the weighting coefficient of the guiding potential function at the current time. The updated weighting coefficients are applied to the guiding potential function, and the synthetic virtual potential field is recalculated to achieve online adaptive adjustment of the potential field structure.

[0012] Furthermore, assigning a threat level value to each projected 3D obstacle point includes: For a 3D point labeled as a dynamic obstacle, its threat value is a weighted sum of a dynamic component that is proportional to the relative approach speed and a static component that is inversely proportional to the relative distance. For a 3D point marked as a static obstacle, its threat value consists only of a static component that is inversely proportional to the relative distance; The relative approach speed is obtained by comparing the rate of change of the centroid position of the three-dimensional point cloud of the same obstacle region in two consecutive frames, and the relative distance is the Euclidean distance from the three-dimensional point to the UAV body.

[0013] Furthermore, the prerequisites and configurations for acquiring the continuous raw image frame sequence of the flight environment captured by the UAV's onboard visual sensor include: Before the UAV takes flight, the intrinsic and extrinsic parameters of the onboard visual sensor have been calibrated. The extrinsic parameter calibration determines the transformation relationship between the visual sensor coordinate system and the UAV body coordinate system. During flight, the onboard computing unit triggers the vision sensor to acquire images at a fixed frequency and ensures that the acquired continuous raw image frame sequence has timestamp information for synchronization with the inertial measurement unit data.

[0014] Compared with the prior art, the beneficial effects of the present invention are: Pixel-level visual analysis of continuous raw image frame sequences acquired by UAV onboard visual sensors can directly identify and segment dynamic and static obstacle regions with potential collision threats, outputting obstacle region segmentation maps. This method can completely preserve the spatial distribution information of obstacles at the image level, distinguish the regional characteristics of different types of obstacles, and present the boundaries and extent of obstacles with pixel-level precision. The correspondence between visual perception information and the actual environment is higher, and the extracted obstacle threat regions can be directly used for subsequent environmental modeling. This ensures that the representation of obstacle information matches the input requirements of obstacle avoidance decisions, improving the accuracy of the visual perception stage in describing environmental threats.

[0015] By simultaneously calculating the real-time pose data of the UAV and spatiotemporally aligning this pose data with the obstacle region segmentation map, a real-time environmental threat field can be generated with the UAV's body coordinate system as a reference. Spatiotemporal alignment eliminates temporal and spatial discrepancies between pose information and obstacle perception information. The construction benchmark of the environmental threat field remains consistent with the UAV's own flight state, and the distribution characteristics of environmental threats can be updated in real-time as the UAV's pose changes. Based on this, a synthetic virtual potential field is constructed, making the distribution of the repulsive and guiding potential fields more closely match the UAV's real-time flight scenario. The theoretical flight velocity vector calculated using the gradient descent method better reflects actual flight conditions. Flight control commands generated after smoothness constraints and dynamic feasibility verification are adapted to the response characteristics of the UAV's flight actuators, directly driving the UAV to complete stable attitude and position adjustments, improving the coherence and reliability of the autonomous obstacle avoidance flight process. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of the vision-based measurement-based autonomous obstacle avoidance flight control method for firefighting drones described in this invention. Figure 2 A flowchart for pixel-level visual analysis to identify dynamic and static obstacle regions; Figure 3A flowchart for synchronously calculating real-time pose data and generating a real-time environmental threat field. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] See Figure 1 This invention provides a vision-based autonomous obstacle avoidance flight control method for firefighting drones. The method includes: acquiring a continuous sequence of raw image frames of the flight environment captured by the drone's onboard visual sensor; performing pixel-level visual analysis on the continuous raw image frame sequence to identify and segment dynamic and static obstacle regions with potential collision threats, and outputting an obstacle region segmentation map; simultaneously calculating the drone's real-time pose data and aligning the real-time pose data with the obstacle region segmentation map in time and space to generate a real-time environmental threat field with reference to the drone's body coordinate system; constructing a synthetic virtual potential field online based on the real-time environmental threat field, comprising a repulsive potential field and a guiding potential field, wherein the repulsive potential field is generated by the obstacle region and the guiding potential field is generated by preset flight waypoints; in the synthetic virtual potential field, iteratively calculating the drone's theoretical flight velocity vector for the next moment using the gradient descent method; performing smoothness constraints and dynamic feasibility checks on the theoretical flight velocity vector for the next moment to generate executable flight control commands; and issuing the executable flight control commands to the drone's flight controller to drive the drone to adjust its attitude and position.

[0019] In one embodiment of the present invention, before the flight mission begins, the UAV has completed the intrinsic and extrinsic parameter calibration of its onboard visual sensor. The extrinsic parameter calibration determines the transformation relationship between the visual sensor coordinate system and the UAV body coordinate system. During flight, the onboard computing unit triggers the visual sensor to acquire images at a fixed frequency and ensures that the acquired continuous raw image frame sequence has timestamp information for synchronization with the inertial measurement unit data. See also... Figure 2For each frame in a continuous sequence of original image frames, a dense optical flow calculation algorithm is applied to obtain the motion vector of each pixel in the image, forming an optical flow field. Based on the optical flow field, a set of pixels with coherent motion patterns is separated using motion vector clustering and region growing techniques, and this set of pixels is labeled as a dynamic obstacle candidate region. Edge detection and texture feature extraction are simultaneously performed on the current image frame. Combined with a prior geometric shape model, contour regions that are significantly different from the background in the image are identified, and these contour regions are labeled as static obstacle candidate regions. The dynamic and static obstacle candidate regions are fused, and noise regions with an area smaller than a preset threshold are filtered out to generate an obstacle region segmentation map containing obstacle category labels and contour information.

[0020] Before flight, the UAV completes the intrinsic and extrinsic parameter calibration of its onboard visual sensor. The extrinsic parameter calibration determines the transformation relationship between the visual sensor coordinate system and the UAV's body coordinate system. This transformation relationship is stored in the onboard computing unit in the form of a rotation matrix and a translation vector. During flight, the onboard computing unit triggers the visual sensor to acquire images at a frequency of 20 Hz, ensuring that the acquired continuous raw image frame sequence has millisecond-level timestamp information. This timestamp is strictly aligned with the sampling time of the inertial measurement unit data for spatiotemporal synchronization of subsequent data.

[0021] In some embodiments, pixel-level visual analysis is performed on a continuous sequence of raw image frames, the core of which is to segment dynamic and static obstacles. For each frame in the continuous sequence of raw image frames, the Farneback dense optical flow calculation algorithm is applied to obtain the motion vector of each pixel in the image between adjacent frames, thereby forming an optical flow field describing the motion of the entire image. Based on the optical flow field, pixel sets with coherent motion patterns are separated through motion vector clustering and region growing techniques. For example, in a fire rescue scene, pixel blocks that move rapidly relative to the building background are separated, and such pixel sets are labeled as candidate regions for dynamic obstacles. Such dynamic obstacles may include flammable debris floating in the air or other moving aircraft. Optionally, Canny edge detection and SIFT texture feature extraction are performed simultaneously on the current image frame. Combined with prior geometric shape models, such as cuboid and cylindrical models, contour regions in the image that are significantly different from the background are identified, such as the window contours of building exteriors or pipe structures protruding from the walls. These contour regions are labeled as candidate regions for static obstacles.

[0022] It is understandable that fusing dynamic and static obstacle candidate regions and filtering out noise regions with an area smaller than a preset threshold are necessary steps in generating a high-quality obstacle region segmentation map. In specific implementations, image morphology closing operations are used to connect and smooth the fused regions. Then, connected regions with a pixel area smaller than 100 pixels are filtered out to eliminate false detections caused by image noise or lighting variations. The final output is an obstacle region segmentation map containing obstacle category labels and precise contour information. In some embodiments, the preset threshold can be adaptively adjusted according to the lighting conditions of the flight environment and the resolution of the visual sensor.

[0023] It is understandable that real-time performance is crucial for visual analysis. In practical implementation, a parallel computing architecture is adopted, distributing optical flow calculation, edge detection, and feature extraction tasks to different computing cores of the onboard graphics processing unit for synchronous execution. This ensures that the entire pixel-level visual analysis process meets the real-time requirements of the control cycle. Optionally, the confidence score for classifying a pixel in an image as a dynamic obstacle is defined as follows: in: Indicates dynamic confidence level. This represents the motion vector of the pixel in the optical flow field. It is the preset maximum motion speed normalization factor. This represents a measure of the consistency of the movement direction of this pixel with that of its surrounding pixels. and These are preset weighting coefficients. Only when... Only when the set threshold is exceeded are the pixels initially included in the dynamic obstacle candidate region.

[0024] In one embodiment of the present invention, see [reference] Figure 3By reading the angular velocity and acceleration data output by the UAV's inertial measurement unit and the position data output by the global navigation satellite system, and using a sensor fusion filter to fuse the angular velocity, acceleration, and position data, the system outputs the UAV's six-degree-of-freedom pose at the current moment. This six-degree-of-freedom pose includes three-dimensional position and three-dimensional attitude. Based on this six-degree-of-freedom pose and the intrinsic and extrinsic parameter matrices of the onboard vision sensor, the obstacle contour points corresponding to each pixel in the obstacle region segmentation image are projected from the image coordinate system to the three-dimensional space under the UAV's body coordinate system. In the body coordinate system, a threat value is assigned to each projected three-dimensional obstacle point. The threat value assignment rule is as follows: for three-dimensional points marked as dynamic obstacles, their threat value is a weighted sum of a dynamic component proportional to the relative approach velocity and a static component inversely proportional to the relative distance; for three-dimensional points marked as static obstacles, their threat value consists only of the static component inversely proportional to the relative distance. The relative approach speed is obtained by comparing the rate of change of the centroid position of the 3D point cloud of the same obstacle region in two consecutive frames, and the relative distance is the Euclidean distance from the 3D point to the UAV body. All 3D point clouds with threat levels together constitute the real-time environmental threat field.

[0025] The specific operation of multi-sensor data fusion and three-dimensional spatial threat quantification involves reading the three-axis angular velocity and three-axis acceleration data output by the UAV's inertial measurement unit (IMU), as well as the latitude, longitude, and altitude position data output by the Global Navigation Satellite System (GNSS). The range of the angular velocity data is ±2000 degrees / second, the range of the acceleration data is ±16g, and the GNSS data update frequency is 10 Hz. In practice, an extended Kalman filter is used as the sensor fusion filter to fuse the angular velocity, acceleration, and position data from the IMU and the GNSS. The filter state variables include the UAV's position, velocity, and attitude quaternions, as well as the IMU's zero bias. The output is the UAV's six-degree-of-freedom pose at the current moment, which includes the three-dimensional position coordinates referenced to the Earth's fixed coordinate system and the three-dimensional attitude represented by quaternions.

[0026] In some embodiments, based on the six-degree-of-freedom pose and the intrinsic and extrinsic parameter matrices of the airborne vision sensor, the obstacle contour points corresponding to each pixel in the obstacle region segmentation map are back-projected from the image coordinate system to the three-dimensional space under the UAV body coordinate system. The intrinsic parameter matrix of the vision sensor includes the focal length and principal point coordinates, and the extrinsic parameter matrix is ​​the rotation matrix and translation vector obtained in advance. The back-projection process first uses the intrinsic parameter matrix to convert the pixel coordinates into the direction vector under the normalized camera coordinate system, and then uses the extrinsic parameter matrix and the rotation matrix corresponding to the current UAV attitude to convert the direction vector to the body coordinate system. Combined with the current position of the UAV and the assumed average height of the obstacle or the coarse depth obtained by stereo vision, the three-dimensional coordinates of the obstacle contour points under the body coordinate system are calculated.

[0027] Assigning a threat value to each projected 3D obstacle point is understandably the core of constructing the environmental threat field. In practice, for 3D points labeled as dynamic obstacles, their threat value is a weighted sum of a dynamic component proportional to the relative approach velocity and a static component inversely proportional to the relative distance. For 3D points labeled as static obstacles, their threat value consists only of the static component inversely proportional to the relative distance. The relative approach velocity is obtained by comparing the rate of change of the centroid position of the 3D point cloud in the same obstacle region across two consecutive frames. The relative distance is the Euclidean distance from the 3D point to the origin of the UAV's coordinate system. Optionally, the formula for calculating the dynamic obstacle threat value can be expressed as: in: Indicates the threat level value. Indicates a scalar value that is relatively close to velocity. It is the preset maximum approach velocity normalization factor. It is a preset safe distance. It is the position vector of the obstacle point in the body coordinate system. This indicates their Euclidean distance, i.e., relative distance. and This is the weighting coefficient for the dynamic component. For static obstacles, the threat level is calculated using the following formula: .

[0028] In some embodiments, the calculation of relative approach speed relies on continuous tracking of the same obstacle. In a specific implementation, for candidate regions of dynamic obstacles labeled with the same ID in consecutive frames, the centroid coordinates of their 3D point clouds are calculated. The centroid coordinates are obtained by calculating the arithmetic mean of the coordinates of all projected points in the region. The centroid movement speed is obtained by subtracting the vector difference between the centroid coordinates of the previous frame and the centroid coordinates of the current frame, and then dividing by the time interval between the two frames. This speed vector is projected onto the unit direction vector pointing from the UAV to the centroid, and the scalar result is the relative approach speed. Optionally, all 3D point clouds with threat levels constitute a real-time environmental threat field. The real-time environmental threat field is stored in the onboard computing unit memory in the form of a 3D voxel grid. Each voxel grid records the sum of all threat levels at its center position. Voxels with a sum exceeding a certain threshold are marked as threat units. It can be understood that the real-time environmental threat field is updated in each control cycle, providing the basic data for the construction of the spatial repulsion force field for the subsequent virtual potential field.

[0029] In one embodiment of the present invention, in a real-time environmental threat field, a repulsive potential function is defined centered on each three-dimensional obstacle point. The value of the repulsive potential function is inversely proportional to the distance from the three-dimensional obstacle point to the current position of the UAV and directly proportional to the threat level carried by the three-dimensional obstacle point. A guiding potential function is defined centered on the spatial position of the next target waypoint set in the UAV flight mission. The value of the guiding potential function is directly proportional to the distance from the current position of the UAV to the target waypoint. On the spatial grid map of the UAV's body coordinate system, the values ​​of the repulsive potential function and the guiding potential function on each spatial grid are vector-superimposed to generate a synthetic virtual potential field. The potential energy gradient direction of the synthetic virtual potential field at any grid indicates the direction of the virtual resultant force that the UAV should experience at the corresponding position.

[0030] In a real-time threat field, a repulsive potential function is defined centered on each 3D obstacle point. The value of the repulsive potential function is inversely proportional to the Euclidean distance from the 3D obstacle point to the drone's current position and directly proportional to the threat level carried by the 3D obstacle point. The design of the repulsive potential function ensures that the closer the obstacle is to the drone or the higher its threat level, the greater the repulsive potential energy generated at the corresponding spatial location. In an example scenario, there is a static obstacle point with a threat level of 0.8 10 meters in front of the drone, and a dynamic obstacle point with a threat level of 1.5 5 meters to the side. Because the dynamic obstacle point is closer and has a higher threat level, it will generate a stronger repulsive potential field around it than the static obstacle point. A specific mathematical expression of the repulsive potential function is as follows: in: This represents the repulsive potential generated by the i-th obstacle point. It is the repulsive potential gain coefficient. It is the threat level value of the i-th obstacle point. It is the Euclidean distance from the drone's current position to the obstacle point. It is a constant greater than zero, used to adjust the rate at which the potential field decays with distance.

[0031] In some embodiments, a guiding potential function is defined centered on the spatial location of the next target waypoint set in the UAV flight mission. The value of the guiding potential function is proportional to the Euclidean distance from the UAV's current position to the target waypoint. The guiding potential function is designed so that the farther the UAV is from the target waypoint, the greater the attractive potential energy generated by the target point, thereby continuously guiding the UAV to move towards the target. On the spatial grid map of the UAV's body coordinate system, the values ​​of the repulsive potential function and the guiding potential function on each spatial grid are vector-superimposed to generate a synthetic virtual potential field. The spatial grid map discretizes a certain range of space around the UAV into a fixed-resolution cubic grid, for example, dividing a 20m × 20m × 10m space around the UAV into cubic grids with 0.2m side lengths. For each spatial grid, its total potential energy... It is obtained by adding the scalar of the repulsive potential of all obstacles at this grid point to the scalar of the guiding potential of the target point, i.e. ,in This represents the value of the guiding potential function. The potential energy gradient direction of the synthesized virtual potential field at any grid indicates the direction of the virtual resultant force that the UAV should experience at the corresponding position. This gradient direction is determined by the spatial partial derivative of the total potential energy function at that point, pointing in the direction of the fastest decrease in potential energy. Under the action of the virtual force, the UAV will move along the direction of the decrease in potential energy.

[0032] It is understandable that the superposition of repulsive and guiding potentials is a vector operation because the gradient of potential energy is a vector. In specific implementation, the repulsive force vector generated by each obstacle point on the center of a certain grid is calculated. This vector is directed from the obstacle point to the grid center, and its magnitude is determined by the negative gradient of the repulsive potential. The guiding force vector generated by the target point on this grid is calculated. This vector is directed from the grid center to the target point, and its magnitude is determined by the negative gradient of the guiding potential. All repulsive and guiding force vectors are geometrically synthesized to obtain the virtual resultant force vector at that grid. The virtual resultant force vector field of all grids constitutes the synthetic virtual potential field used for path planning. Optionally, the table below shows a simplified example of calculating the potential field values ​​of grids at different locations in a two-dimensional plane under the action of a single obstacle and a single target point, where the obstacle is located at coordinates (0,0) and the threat level is... The target point is located at coordinates (10,0). , Guiding potential coefficient , , For the distance to the target point, please refer to Table 1.

[0033] Table 1: Comparison Table of Numerical Calculation of Synthetic Virtual Potential Field In some embodiments, the scale and resolution of the spatial raster map can be adjusted according to computing resources and the complexity of the flight environment. Lower resolution can be used in open areas to improve computational speed, while higher resolution is used in areas with dense obstacles to obtain a more accurate potential field description. It is understood that the synthesized virtual potential field is dynamically updated. As the UAV moves, new obstacles are detected, or the target waypoint is updated, the potential field needs to be recalculated. Online construction means that a potential field construction needs to be completed once in each control cycle based on the latest real-time environmental threat field and the UAV's current position. Optionally, for 3D obstacle points that are too far from the UAV's current position or have extremely low threat values, they can be ignored when calculating the repulsive potential to further reduce computational load.

[0034] In one embodiment of the present invention, the position of the UAV in its body coordinate system at the current moment is used as the starting point for the gradient descent method iteration in the synthetic virtual potential field. The potential energy gradient of this starting point in the synthetic virtual potential field is calculated; this potential energy gradient is a three-dimensional vector. The inverse direction of the potential energy gradient is normalized, and the resulting direction is taken as the expected movement direction of the UAV at its current position. The expected movement direction is multiplied by a reference velocity scalar dynamically adjusted according to the distance of the UAV to the target waypoint to obtain the theoretical flight velocity vector at the next moment. When performing smoothness constraints and dynamic feasibility verification on the theoretical flight velocity vector at the next moment, the actual flight velocity vector of the UAV in the previous control cycle needs to be obtained. The difference between the theoretical flight velocity vector and the actual flight velocity vector at the next moment is calculated, and a low-pass filter is applied to this difference to smooth the abrupt changes in the velocity command. The smoothed velocity vector is input into the UAV's inverse dynamic model, which calculates the rotor thrust required to achieve the smoothed velocity vector change based on the UAV's mass, moment of inertia, maximum thrust, and torque parameters. The calculation of each rotor thrust is verified to be within the range of thrust that the motor can provide. If it exceeds the range, the thrust is limited to the maximum value, and the corresponding achievable velocity vector is recalculated. This velocity vector and its corresponding attitude angle command together constitute the executable flight control command.

[0035] In the synthetic virtual potential field, the gradient descent method is used to iteratively calculate the theoretical flight velocity vector of the UAV at the next moment. The UAV's current position in the body coordinate system is used as the iteration starting point for the gradient descent method. The potential energy gradient in the synthetic virtual potential field at the iteration starting point is calculated. The potential energy gradient is a three-dimensional vector obtained by calculating the partial derivatives of the synthetic virtual potential field function along the X, Y, and Z coordinate axes at the iteration starting point. The inverse direction of the potential energy gradient is normalized, and the resulting direction is taken as the expected movement direction of the UAV at the current position. The expected movement direction is a unit vector indicating the direction of the fastest potential energy decrease. Multiplying the expected movement direction by a reference velocity scalar dynamically adjusted according to the distance from the UAV to the target waypoint yields the theoretical flight velocity vector at the next moment. (Reference velocity scalar) The calculation follows the formula: in: It is the preset minimum flight speed. It is the preset maximum flight speed. It is the attenuation coefficient. This is the Euclidean distance from the drone's current position to the target waypoint. When the drone moves away from the target, the base speed approaches its maximum value to quickly close the distance; when the drone approaches the target, the base speed decreases to facilitate precise hovering.

[0036] In some embodiments, the theoretical flight velocity vector at the next moment is subjected to smoothness constraints and dynamic feasibility verification to generate executable flight control commands. The actual flight velocity vector of the UAV in the previous control cycle is obtained; this actual flight velocity vector is estimated by data fusion from the inertial measurement unit and the global navigation satellite system. The difference between the theoretical and actual flight velocity vectors at the next moment is calculated, and a low-pass filter is applied to this difference to smooth abrupt changes in the velocity command. The low-pass filter can be a first-order inertial element with a transfer function of... ,in This is the time constant, used to control the smoothness level; see Table 2.

[0037] Table 2: Speed ​​Command Smoothing Filter Parameter Settings Table It is understandable that inputting the smoothed velocity vector into the UAV's inverse dynamics model is a crucial step in generating the underlying control commands. The inverse dynamics model calculates the thrust required for each rotor to achieve the smoothed velocity vector change based on the UAV's mass, moment of inertia, maximum thrust, and torque parameters. For a typical "+"-shaped quadcopter UAV, its inverse dynamics model needs to calculate the thrust that each of the four rotors should generate, based on the desired body coordinate system acceleration and angular acceleration, combined with the current attitude. The calculated rotor thrust is then checked to ensure it is within the range of thrust provided by the motors. The rotor thrust range is determined by the performance parameters of the motors and propellers, and there exists a minimum thrust and a maximum thrust. If any calculated rotor thrust exceeds the maximum value, all rotor thrusts are proportionally limited to the maximum value range, and the corresponding achievable velocity vector and UAV attitude are recalculated using the limited thrust. Optionally, the recalculation process is based on the UAV's forward dynamics model, where, given the limited thrust and current state, forward integration is used to solve for the achievable velocity and attitude of the UAV in the next control cycle. Ultimately, the recalculated achievable velocity vector and its corresponding attitude angle commands together constitute executable flight control commands, including roll, pitch, and yaw angles.

[0038] In some embodiments, the time constant of the low-pass filter can be dynamically adjusted according to the environmental congestion level. In a specific implementation, when the real-time environmental threat field detects a high density of nearby obstacles, the time constant is increased. To obtain smoother commands and reduce oscillations caused by frequent acceleration and deceleration of the drone; to reduce the time constant when the environment is open. To improve command response speed. It's understandable that dynamic feasibility verification is the final guarantee for safe flight, ensuring that the calculated control commands do not exceed the limits of the UAV's physical actuators. Optionally, when calculating rotor thrust, a thrust margin is reserved to account for unmodeled dynamics such as model errors or external wind disturbances. The final flight control commands issued have velocity vectors that have been smoothed and verified for feasibility, and attitude angle commands that represent the target attitude required to achieve that velocity vector.

[0039] In one embodiment of the present invention, after generating executable flight control commands, the flight status is continuously monitored and the potential field is updated online. After executing the current control command, a new continuous sequence of original image frames and real-time pose data of the UAV are acquired again. Based on the new data, the steps from visual analysis to generating a real-time environmental threat field are repeated, and the real-time environmental threat field is updated. In the synthesized virtual potential field, the weight coefficients of the guiding potential function are dynamically adjusted according to the current position of the UAV and the target waypoint position, so that the repulsive potential field dominates when close to obstacles, and the guiding potential field dominates when in open space. The specific steps for dynamically adjusting the weight coefficients of the guiding potential function are as follows: calculate the ratio of the threat level of the nearest obstacle to the UAV in the current real-time environmental threat field to the distance, as an environmental congestion index. Define a smoothing function that is negatively correlated with the environmental congestion index, and use the output value of the smoothing function as the weight coefficient of the guiding potential function at the current moment. Apply the updated weight coefficients to the guiding potential function and recalculate the synthesized virtual potential field to achieve online adaptive adjustment of the potential field structure.

[0040] After executing the current control command, the flight control cycle enters the next cycle, reacquiring a new continuous sequence of raw image frames captured by the onboard vision sensor and real-time UAV pose data calculated by the sensor fusion filter. Based on the new continuous sequence of raw image frames and the new real-time UAV pose data, the complete steps from pixel-level visual analysis to generating a real-time environmental threat field are repeated, thereby updating the real-time environmental threat field. The updated real-time environmental threat field reflects the latest perceived distribution of environmental obstacles and threat levels. In the synthesized virtual potential field, the weight coefficients of the guiding potential function are dynamically adjusted according to the UAV's current position and the target waypoint position. The purpose of this dynamic adjustment is to enable the UAV to adapt to environmental changes during flight, allowing the repulsive potential field to dominate for priority obstacle avoidance when approaching obstacles, and the guiding potential field to dominate for efficient progress towards the target waypoint in open space.

[0041] In some embodiments, dynamically adjusting the weighting coefficients of the guiding potential function includes a specific calculation process: calculating the ratio of the threat level of the nearest obstacle to the drone in the current real-time environmental threat field to its distance; using this ratio as an environmental congestion index, which is a dimensionless number; a larger value indicates a more congested local environment and a higher threat level for the drone. A smoothing function negatively correlated with the environmental congestion index is defined, and the output value of the smoothing function is used as the weighting coefficients of the guiding potential function at the current moment. Optionally, the smoothing function adopts an exponential decay form, and its mathematical expression is: in: These are the calculated weighting coefficients of the guiding potential function. It is the preset maximum value of the weighting coefficients of the guiding potential function. It is the decay constant used to control the rate at which the weight decreases with increasing environmental crowding. This is a calculated environmental congestion index. When the environmental congestion index... When the weighting coefficient increases, This reduces the influence of the guiding potential field on the synthesized virtual potential field, allowing the repulsive potential field to become relatively dominant.

[0042] It can be understood that applying the updated weighting coefficients to the guiding potential function and recalculating the synthesized virtual potential field achieves online adaptive adjustment of the potential field structure. In specific implementations, the general form of the guiding potential function is: ,in It is a distance function. This refers to the distance from the drone to the target waypoint, and the weighting coefficients in the formula are updated online and adaptively in real time. Each time the real-time environmental threat field is updated, the weighting coefficients need to be recalculated based on the latest environmental congestion index. and use new The composite virtual potential field on the entire spatial raster map is recalculated. In an example scenario, a drone flies from open airspace toward a building whose window is marked as a static obstacle. When the distance to the building is relatively far, the environmental congestion index is low, and the weighting coefficient is low. Maintaining a high level and a strong guiding potential field, the UAV flies rapidly in a straight line towards the target waypoint; when approaching buildings, the threat level of window obstacles and the very close distance cause the environmental congestion index to increase sharply, and the weighting coefficient... By rapidly reducing the smoothing function, the guiding potential field is weakened, while the repulsive potential field generated by the obstacle is relatively strengthened. The equipotential lines of the synthesized virtual potential field bend significantly in front of the obstacle, thereby guiding the drone to fly around the window.

[0043] In some embodiments, the calculation of the environmental congestion index can consider the combined impact of multiple near-end obstacles, rather than just the nearest obstacle. In specific implementations, the weighted sum of the threat level to distance ratios of all obstacles in the real-time environmental threat field that exceed a threshold and are within a preset range of the drone can be calculated as the environmental congestion index. This provides a more comprehensive reflection of the local environmental congestion situation. It is understood that the weighting coefficients... The update frequency is consistent with the control loop frequency, ensuring that the synthetic virtual potential field used for path planning in each control cycle is based on the latest environmental perception information and UAV status. Optionally, to prevent the weighting coefficients from frequently oscillating at obstacle boundaries, the calculated environmental congestion index can be adjusted. A first-order low-pass filter is applied, and then the filtered values ​​are substituted into a smoothing function to calculate the weighting coefficients, thereby achieving a smooth transition of the potential field structure. The online-updated real-time environmental threat field and the adaptively adjusted synthetic virtual potential field enable the UAV's obstacle avoidance behavior to dynamically respond to environmental changes.

[0044] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A vision-based measurement-based autonomous obstacle avoidance flight control method for firefighting drones, characterized in that, Includes the following steps: Acquire a continuous sequence of raw image frames of the flight environment captured by the UAV's onboard visual sensor; Pixel-level visual analysis is performed on the continuous raw image frame sequence to identify and segment dynamic obstacle regions and static obstacle regions with potential collision threats, and an obstacle region segmentation map is output. The instantaneous pose data of the UAV is calculated synchronously, and the instantaneous pose data is spatiotemporally aligned with the obstacle area segmentation map to generate a real-time environmental threat field with the UAV body coordinate system as a reference. Based on the real-time environmental threat field, a synthetic virtual potential field with a repulsive potential field and a guiding potential field is constructed online, wherein the repulsive potential field is generated by the obstacle region and the guiding potential field is generated by the preset flight waypoint; In the synthetic virtual potential field, the gradient descent method is used to iteratively calculate the theoretical flight velocity vector of the UAV at the next moment; The smoothness constraint and dynamic feasibility verification of the theoretical flight velocity vector at the next moment are performed to generate executable flight control commands; The executable flight control commands are sent to the UAV's flight controller to drive the UAV to complete attitude and position adjustments.

2. The autonomous obstacle avoidance flight control method for firefighting drones based on vision measurement according to claim 1, characterized in that, Perform pixel-level visual analysis on the continuous raw image frame sequence to identify and segment dynamic and static obstacle regions with potential collision threats, including: For each frame in the continuous original image frame sequence, a dense optical flow calculation algorithm is applied to obtain the motion vector of each pixel in the image, forming an optical flow field; Based on the optical flow field, a set of pixels with a coherent motion pattern is separated by motion vector clustering and region growing techniques, and the set of pixels is labeled as a candidate region for dynamic obstacles. Edge detection and texture feature extraction are performed simultaneously on the current image frame. Combined with the prior geometric shape model, contour regions that are significantly different from the background in the image are identified, and these contour regions are marked as static obstacle candidate regions. The dynamic obstacle candidate region and the static obstacle candidate region are merged, and noise regions with an area smaller than a preset threshold are filtered out to generate an obstacle region segmentation map containing obstacle category labels and contour information.

3. The autonomous obstacle avoidance flight control method for firefighting drones based on vision measurement according to claim 2, characterized in that, The simultaneous calculation of the UAV's real-time pose data, and the spatiotemporal alignment of the real-time pose data with the obstacle region segmentation map, generates a real-time environmental threat field with the UAV's body coordinate system as a reference, including: Read the angular velocity and acceleration data output by the UAV's inertial measurement unit, as well as the position data output by the global navigation satellite system; The angular velocity, acceleration, and position data are fused and calculated using a sensor fusion filter to output the six-degree-of-freedom pose of the UAV at the current moment, which includes three-dimensional position and three-dimensional attitude. Based on the six-degree-of-freedom pose and the intrinsic and extrinsic parameter matrices of the airborne vision sensor, the obstacle contour points corresponding to each pixel in the obstacle region segmentation image are back-projected from the image coordinate system to the three-dimensional space under the UAV body coordinate system. In the body coordinate system, a threat value is assigned to each projected 3D obstacle point. The threat value is calculated based on the obstacle type and its relative distance from the drone. All 3D point clouds with threat values ​​together constitute the real-time environmental threat field.

4. The autonomous obstacle avoidance flight control method for firefighting drones based on vision measurement according to claim 3, characterized in that, Based on the real-time environmental threat field, a synthetic virtual potential field with a repulsive potential field and a guiding potential field is constructed online, including: In the real-time environmental threat field, a repulsion potential function is defined with each three-dimensional obstacle point as the center. The value of the repulsion potential function is inversely proportional to the distance from the three-dimensional obstacle point to the current position of the UAV and directly proportional to the threat level carried by the three-dimensional obstacle point. A guiding potential function is defined with the spatial position of the next target waypoint set in the UAV flight mission as the center. The value of the guiding potential function is proportional to the distance from the current position of the UAV to the target waypoint. On the spatial grid map of the UAV's body coordinate system, the values ​​of the repulsive potential function and the guiding potential function on each spatial grid are vector-superimposed to generate the synthetic virtual potential field. The potential gradient direction of the synthetic virtual potential field at any grid indicates the direction of the virtual resultant force that the UAV should experience at the corresponding position.

5. The autonomous obstacle avoidance flight control method for firefighting drones based on vision measurement according to claim 4, characterized in that, In the synthesized virtual potential field, the theoretical flight velocity vector of the UAV at the next moment is iteratively calculated using the gradient descent method, including: The current position of the UAV in the body coordinate system is used as the starting point for the gradient descent method iteration; Calculate the potential energy gradient of the starting point of the iteration in the synthetic virtual potential field, where the potential energy gradient is a three-dimensional vector; The opposite direction of the potential energy gradient is normalized, and the resulting direction is taken as the expected movement direction of the UAV at the current position. The desired direction of movement is multiplied by a baseline velocity scalar that is dynamically adjusted based on the distance from the UAV to the target waypoint to obtain the theoretical flight velocity vector for the next moment.

6. The autonomous obstacle avoidance flight control method for firefighting drones based on vision measurement according to claim 5, characterized in that, The theoretical flight velocity vector at the next moment is subjected to smoothness constraints and dynamic feasibility verification to generate executable flight control commands, including: Obtain the actual flight speed vector of the UAV in the previous control cycle; Calculate the difference between the theoretical flight speed vector and the actual flight speed vector at the next moment, and apply a low-pass filter to the difference to smooth the sudden changes in speed command; The smoothed velocity vector is input into the inverse dynamics model of the UAV. The inverse dynamics model calculates the thrust of each rotor required to achieve the change of the smoothed velocity vector based on the mass, moment of inertia, maximum thrust and torque parameters of the UAV. The calculated thrust of each rotor is verified to be within the range of thrust that the motor can provide. If it exceeds the range, the thrust is limited to the maximum value, and the corresponding achievable velocity vector is recalculated. The velocity vector and its corresponding attitude angle command together constitute the executable flight control command.

7. The autonomous obstacle avoidance flight control method for firefighting drones based on vision measurement according to claim 6, characterized in that, It also includes the steps of continuously monitoring the flight status and updating the potential field online after generating executable flight control commands: After executing the current control command, a new continuous sequence of raw image frames and real-time pose data of the UAV are acquired again. Based on the new data, the steps from visual analysis to generating a real-time environmental threat field are repeated, and the real-time environmental threat field is updated. In the synthesized virtual potential field, the weighting coefficients of the guiding potential function are dynamically adjusted according to the current position of the UAV and the target waypoint position, so that the repulsive potential field dominates when approaching an obstacle, and the guiding potential field dominates when in open space.

8. The autonomous obstacle avoidance flight control method for firefighting drones based on vision measurement according to claim 7, characterized in that, The dynamic adjustment of the weighting coefficients of the guiding potential function includes: Calculate the ratio of the threat level of the nearest obstacle to the drone in the current real-time environmental threat field to its distance, and use it as an indicator of environmental congestion. Define a smoothing function that is negatively correlated with the environmental congestion index, and use the output value of the smoothing function as the weighting coefficient of the guiding potential function at the current time. The updated weighting coefficients are applied to the guiding potential function, and the synthetic virtual potential field is recalculated to achieve online adaptive adjustment of the potential field structure.

9. The autonomous obstacle avoidance flight control method for firefighting drones based on vision measurement according to claim 8, characterized in that, Assigning a threat level value to each projected 3D obstacle point includes: For a 3D point labeled as a dynamic obstacle, its threat value is a weighted sum of a dynamic component that is proportional to the relative approach speed and a static component that is inversely proportional to the relative distance. For a 3D point marked as a static obstacle, its threat value consists only of a static component that is inversely proportional to the relative distance; The relative approach speed is obtained by comparing the rate of change of the centroid position of the three-dimensional point cloud of the same obstacle region in two consecutive frames, and the relative distance is the Euclidean distance from the three-dimensional point to the UAV body.

10. The autonomous obstacle avoidance flight control method for firefighting drones based on vision measurement according to claim 9, characterized in that, The prerequisites and configurations for acquiring the continuous raw image frame sequence of the flight environment captured by the UAV's onboard visual sensor include: Before the UAV takes flight, the intrinsic and extrinsic parameters of the onboard visual sensor have been calibrated. The extrinsic parameter calibration determines the transformation relationship between the visual sensor coordinate system and the UAV body coordinate system. During flight, the onboard computing unit triggers the vision sensor to acquire images at a fixed frequency and ensures that the acquired continuous raw image frame sequence has timestamp information for synchronization with the inertial measurement unit data.

Images