Unmanned aerial vehicle adaptive path planning method and device based on multi-resolution semantic segmentation, equipment and storage medium

Abstract

The invention provides an unmanned aerial vehicle adaptive path planning method and device based on multi-resolution semantic segmentation, equipment and a storage medium, and relates to the technical field of unmanned aerial vehicle path planning. According to the method, a GCN is selected as a to-be-trained model; for a target area, marking topographic data of the target area, and training the GCN by using the topographic data to obtain a trained model; calculating an initial path for guiding the unmanned aerial vehicle to fly at a fixed height above the target area based on a zigzag mower strategy; the unmanned aerial vehicle flies according to the initial path and obtains a current aerial image every time the unmanned aerial vehicle arrives at an aerial photography point; semantic segmentation of pixels is carried out on the current aerial image; taking the waypoint of the unmanned aerial vehicle at the current moment as input, and outputting an aerial photography point at the next moment according to the semantic segmentation result; and repeating the process to obtain a corrected unmanned aerial vehicle path. According to the method, accurate semantic segmentation results can be provided at different flight heights, so that the path planning and data acquisition efficiency of the unmanned aerial vehicle is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle path planning, and in particular to a method, device, equipment and storage medium for unmanned aerial vehicle adaptive path planning based on multi-resolution semantic segmentation. Background Art

[0002] Drones are increasingly being used in remote sensing and monitoring, especially in agriculture, environmental monitoring, and disaster management. Drones can quickly and inexpensively acquire high-resolution image data, providing valuable information for a variety of applications.

[0003] However, when performing high-resolution data acquisition in a large-scale environment, flight time and battery life become major limiting factors.

[0004] Traditional path planning methods are usually pre-set and cannot be adjusted according to real-time data, which may lead to a waste of resources. Therefore, how to maximize the value of data within a limited flight time becomes a key issue. Summary of the invention

[0005] Purpose of the invention: To propose a method, device, equipment and storage medium for UAV adaptive path planning based on multi-resolution semantic segmentation to solve the above-mentioned problems existing in the prior art.

[0006] First, a UAV adaptive path planning method based on multi-resolution semantic segmentation is proposed, which includes the following steps:

[0007] Select an open source graph convolutional network (GCN) as the model to be trained; for a target area, mark its terrain data, use the terrain data to train the GCN, and obtain a trained model;

[0008] An initial path for directing the UAV flight is calculated based on a standard sawtooth mower strategy at a fixed height above the target area;

[0009] The UAV flies along the initial path, and acquires the current aerial image each time the UAV reaches the aerial photography point; and performs pixel semantic segmentation on the current aerial image;

[0010] The waypoint of the drone’s current location is used as input, and the aerial photography point of the next moment is output according to the semantic segmentation result; this process is repeated to obtain the corrected drone path.

[0011] In a further embodiment of the first aspect, when marking the terrain data of the target area, a camera is used to observe the same area from different heights, and the result obtained at the highest resolution is retained.

[0012] In a further embodiment of the first aspect, an initial path for directing the drone flight is calculated based on a standard sawtooth mower strategy at a fixed height above the target area. in This altitude is the highest altitude of the current waypoint;

[0013] At each waypoint w i There are two situations for drones:

[0014] Case 1: Move to w following the precomputed coverage path i+1 ;

[0015] The second case: Check the current area at an altitude lower than the predetermined value;

[0016] In the second case, define the second lawnmower path W h′ ={w 0 ,w 1 ,...w n}.

[0017] In a further embodiment of the first aspect, the waypoint of the drone at the current moment is used as the input of the decision function, and the aerial point w at the next moment is output according to the semantic segmentation result. i+1 , or 0 ∈W h′ ;

[0018] If the height changes, the decision function also outputs the required height h ′ ;

[0019] Define a subset of a set of class labels For the segmented image, calculate the proportion of pixels belonging to these labels to the total number of pixels:

[0020]

[0021] Among them, P tot is the total number of pixels in the image, p l is classified as The higher the semantic segmentation ratio σ, the more misclassifications there are in the category label.

[0022] Aerial photography height difference Δh=h max -h ′ , the corresponding semantic segmentation ratio The accuracy of σ is compared with the predicted segmentation by calculating the mean intersection over union (mIoU), which is the semantic ratio between the intersection of the ground truth data and the predicted segmentation data and the union of the same number, expressed as:

[0023]

[0024] In the formula, gt l represents the ground truth data, pred l Represents the predicted segmentation data.

[0025] Consider two sets of observations, representing the relationship between the semantic ratio σ and the drone height h And the relationship between the semantic ratio σ and mIoU Υ, respectively, is expressed as:

[0026]

[0027] Use DGP regression to fit these two sets of observations; use the ground sampling distance GSD to determine the image resolution for semantic segmentation:

[0028]

[0029] In the formula, h represents the height of the drone, S w represents the width of the camera sensor, f represents the focal length of the camera, and I w Indicates the width of the image.

[0030] In a further embodiment of the first aspect, each time the drone decides to lower its altitude, a new Δσ is calculated. ′ and Δh ′ , and re-use DGP regression to fit and Υ observation data; this process is repeated at each waypoint of the original lawn mower path, and the corrected drone path is finally obtained.

[0031] According to a second aspect of the present invention, an adaptive path planning device for a UAV is provided, which includes a model training unit, a first calculation unit, a second calculation unit, and a path correction unit.

[0032] The model training unit selects an open source graph convolutional network GCN as the model to be trained; for a target area, its terrain data is marked, and the GCN is trained using the terrain data to obtain a trained model.

[0033] The first computing unit is used to calculate an initial path for guiding the UAV to fly based on a standard sawtooth mower strategy at a fixed height above the target area.

[0034] The second computing unit is used to obtain a current aerial image each time the drone arrives at an aerial photography point when flying along the initial path; and perform pixel semantic segmentation on the current aerial image.

[0035] The path correction unit takes the waypoint of the drone at the current moment as input and outputs the aerial photography point at the next moment according to the semantic segmentation result; this process is repeated to obtain the corrected drone path.

[0036] In a third aspect of the present invention, an electronic device is proposed, which includes a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for adaptive path planning of a drone based on multi-resolution semantic segmentation as described in the first aspect is implemented.

[0037] In a fourth aspect of the present invention, a computer-readable storage medium is proposed, which stores at least one executable instruction. When the executable instruction is executed on an electronic device, the electronic device executes the drone adaptive path planning method based on multi-resolution semantic segmentation as described in the first aspect.

[0038] Beneficial effects: The UAV adaptive path planning method based on multi-resolution semantic segmentation proposed in the present invention can adjust the flight path in real time according to the images taken by the UAV to obtain high-resolution semantic segmentation results. The algorithm pays special attention to adjusting the flight altitude of the UAV for close inspection when a detailed area is detected, thereby avoiding high-resolution comprehensive scanning of the entire area, saving energy and time. In addition, the present invention introduces a new accuracy model to capture the relationship between the flight altitude of the UAV and the accuracy of semantic segmentation. Based on a deep learning architecture, the model can provide accurate semantic segmentation results at different flight altitudes, thereby optimizing the path planning and data collection efficiency of the UAV. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 4 is a flow chart of a method for adaptive path planning of a UAV proposed in an embodiment of the present invention.

[0040] Figure 2 It is a structural schematic diagram of a UAV adaptive path planning device proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described.

[0042] In recent years, the application of drones in remote sensing and monitoring has increased rapidly due to their high mobility, low cost and flexible deployment. However, the limitations of flight time and battery life make it difficult to obtain high-resolution image data over a large area. Traditional path planning methods are usually pre-set and cannot be adjusted according to real-time data, which may lead to a waste of resources.

[0043] Therefore, the present invention proposes an adaptive path planning algorithm that can dynamically adjust the flight path of the drone according to the image data acquired in real time. When the drone detects that there is an area with rich details in the image, it will lower the flight altitude to obtain a high-resolution image; while in areas with less details, it will maintain a higher flight altitude to save energy and time. In this way, the drone can concentrate resources on areas where high-resolution data is required within a limited flight time, thereby improving the efficiency and accuracy of overall data acquisition. This method is particularly suitable for application scenarios such as agriculture, environmental monitoring, and disaster management that require large-scale and efficient data acquisition. In addition, the present invention also introduces a new accuracy model, which is based on a deep learning architecture and can capture the relationship between the drone's flight altitude and semantic segmentation accuracy. Through this model, the drone can evaluate the resolution requirements of the current image in real time during flight, thereby optimizing path planning. This not only improves the accuracy of semantic segmentation, but also effectively saves flight time and energy. Therefore, the study of the drone's adaptive path planning method for multi-resolution semantic segmentation is particularly important.

[0044] This embodiment proposes a multi-resolution semantic segmentation UAV adaptive path planning method, which can dynamically adjust the flight path of the UAV according to the real-time acquired image data. The processing flow of this method is shown in Figure 1 , as follows:

[0045] Model initialization: We use previously labeled terrain data to empirically determine how classification accuracy varies with altitude. We use GCN trained at different altitudes to enable it to generalize across different altitudes without retraining. If the camera observes the same area from different altitudes, we retain the results obtained at the highest resolution. We initialize the decision function based on DGP. We then initialize the path planning strategy with GCN and DGP decision function.

[0046] Build a multi-resolution semantic segmentation module: Based on the lightweight ERFNet architecture, each time the drone arrives at the aerial photography point, the pixel semantic segmentation of the current aerial image is performed, and the pixel semantic segmentation is performed on the set A category label is assigned to each pixel in , where C is the number of categories. For 3D mapping based on image information, the distance measurement is weighted according to the resolution, and the decision function is updated online based on accurate pixel-level semantic segmentation in dense images to achieve more reliable prediction planning performance without relying on sensor models for path planning.

[0047] Initialization Path Strategy: The initial UAV flight path is calculated based on a standard sawtooth mower strategy at a fixed altitude above the target area, which can effectively cover the area based on a series of waypoints w i Defines a lawnmower path in This altitude is the highest altitude of the current waypoint. i At this point, the drone decides to follow the pre-computed coverage path, i.e., move to w i+1 , or examine the current area more carefully at a lower altitude. In the second case, define a second mower path W h′ ={w 0 ,w 1 ,...w n}.

[0048] Decision function initialization: This function takes a given waypoint as input and outputs the next aerial point based on the semantic segmentation result or 0 ∈W h∈ If the height changes, the decision function also outputs the required height h ′ In addition, we define a subset of class labels These labels are useful for more detailed semantic analysis. For the segmented image, calculate the proportion of pixels belonging to these labels to the total number of pixels: Among them, P tot is the total number of pixels in the image, p l is classified as The total number of pixels with the label in the image. A higher semantic segmentation ratio σ means that there may be more misclassifications in the category label. Further quantification, the aerial height difference Δh = h max -h ′ , the corresponding semantic segmentation ratio The accuracy of σ is compared with the predicted segmentation by calculating the mean intersection over union (mIoU). mIoU is the semantic ratio between the intersection between the ground truth data (gt) and the predicted segmentation (pred) and the union of the same number, expressed as:

[0049]

[0050] ΔmIoU is expressed as the difference in semantic ratios at different heights Further consider two sets of observations, representing the relationship between the semantic ratio σ and the drone height h And the relationship between semantic ratio σ and mIoU Υ, respectively expressed as DGP regression is used to fit these two sets of observations. The ground sample distance (GSD) is used to determine the image resolution for semantic segmentation. h represents the altitude of the drone, S w represents the width of the camera sensor, f represents the focal length of the camera, and I w Indicates the width of the image.

[0051] Decision function update: Every time the drone decides to lower its altitude, a new Δσ is calculated ′ and Δh ′ , and re-use DGP regression to fit and Υ observation data. This process is repeated for each waypoint on the original mower path.

[0052] See Figure 2 , this embodiment also proposes an unmanned aerial vehicle adaptive path planning device 500, which includes a model training unit 501, a first computing unit 502, a second computing unit 503, and a path correction unit 504. The model training unit 501 selects an open source graph convolutional network GCN as a model to be trained; for a target area, its terrain data is marked, and the GCN is trained using the terrain data to obtain a trained model. The first computing unit 502 is used to calculate an initial path for guiding the flight of the unmanned aerial vehicle based on a standard sawtooth mower strategy at a fixed height above the target area. The second computing unit 503 is used to obtain a current aerial image each time the unmanned aerial vehicle arrives at an aerial photography point when the unmanned aerial vehicle flies according to the initial path; and perform pixel semantic segmentation on the current aerial image. The path correction unit 504 takes the waypoint at the current moment of the unmanned aerial vehicle as input, and outputs the aerial photography point at the next moment according to the semantic segmentation result; repeat this process to obtain a corrected unmanned aerial vehicle path.

[0053] The UAV adaptive path planning device 500 is an electronic device that automatically executes the process of the UAV adaptive path planning method based on multi-resolution semantic segmentation disclosed in the above embodiment. The electronic device includes a processor, a memory, a communication interface and a communication bus. The processor, the memory and the communication interface communicate with each other through the communication bus. The memory is used to store at least one executable instruction, which enables the processor to execute Figure 1 The specific process of the UAV adaptive path planning method based on multi-resolution semantic segmentation disclosed in the above embodiment will not be repeated here.

[0054] In addition, in the actual application process of the UAV adaptive path planning method based on multi-resolution semantic segmentation disclosed in the above embodiment, its operating logic can be written into a set of executable instructions, and the executable instructions can be written into a storage medium and run on an electronic device.

[0055] More specific examples of the computer-readable storage medium mentioned in this embodiment may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device.

[0056] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for adaptive path planning of unmanned aerial vehicles based on multi-resolution semantic segmentation, characterized in that: The steps include: Select an open source graph convolutional network (GCN) as the model to be trained; for a target area, mark its terrain data, use the terrain data to train the GCN, and obtain a trained model; An initial path for directing the UAV flight is calculated based on a standard sawtooth mower strategy at a fixed height above the target area; The UAV flies along the initial path, and acquires the current aerial image each time the UAV reaches an aerial photography point; Perform pixel semantic segmentation on the current aerial image; Taking the current waypoint of the drone as input, the next waypoint is output based on the semantic segmentation result. Repeat this process to obtain the corrected drone path.

2. The method for adaptive path planning of unmanned aerial vehicles based on multi-resolution semantic segmentation according to claim 1, characterized in that: When marking the terrain data of the target area, use cameras to observe the same area from different heights and keep the results obtained at the highest resolution.

3. The method for adaptive path planning of unmanned aerial vehicles based on multi-resolution semantic segmentation according to claim 1, characterized in that: An initial path is calculated to direct the drone based on a standard sawtooth mower strategy at a fixed height above the target area. in This altitude is the highest altitude of the current waypoint; At each waypoint w i There are two situations for drones: Case 1: Move to w following the precomputed coverage path i+1 ; The second case: Check the current area at an altitude lower than the predetermined value; In the second case, define the second lawnmower path W h′ ={w0,w1,...w n }.

4. The method for adaptive path planning of unmanned aerial vehicles based on multi-resolution semantic segmentation according to claim 3 is characterized in that: The current waypoint of the drone is used as the input of the decision function, and the next aerial point w is output according to the semantic segmentation result. i+1 , Or w0∈W h′ ; If the height changes, the decision function also outputs the required height h ′ ; Define a subset of a set of class labels For the segmented image, calculate the proportion of pixels belonging to these labels to the total number of pixels: Among them, P tot is the total number of pixels in the image, p l is the total number of pixels classified as labels in l; the higher the semantic segmentation ratio σ, the more misclassifications there are in the category label.

5. The method for adaptive path planning of unmanned aerial vehicles based on multi-resolution semantic segmentation according to claim 4, characterized in that: Aerial photography height difference Δh=h max -h ′ , the corresponding semantic segmentation ratio The accuracy of σ is compared with the predicted segmentation by calculating the mean intersection over union (mIoU), which is the semantic ratio between the intersection of the ground truth data and the predicted segmentation data and the union of the same number, expressed as: In the formula, gt l represents the ground truth data, pred l Represents the predicted segmentation data.

6. The method for adaptive path planning of unmanned aerial vehicles based on multi-resolution semantic segmentation according to claim 5, characterized in that: Also includes: Consider two sets of observations, representing the relationship θ between the semantic ratio σ and the drone height h, and the relationship Υ between the semantic ratio σ and mIoU, respectively, as: Use DGP regression to fit these two sets of observations; use the ground sampling distance GSD to determine the image resolution for semantic segmentation: In the formula, h represents the height of the drone, S w represents the width of the camera sensor, f represents the focal length of the camera, and I w Indicates the width of the image.

7. The method for adaptive path planning of unmanned aerial vehicles based on multi-resolution semantic segmentation according to claim 6, characterized in that: Every time the drone decides to lower its altitude, it calculates a new Δσ ′ and Δh ′ , and re-use DGP regression to fit the θ and Υ observations; This process is repeated at each waypoint of the original lawnmower path, ultimately resulting in a corrected drone path.

8. An adaptive path planning device for a drone, characterized in that: include: Model training unit; The model training unit selects an open source graph convolutional network GCN as a model to be trained; For a target area, mark its terrain data, use the terrain data to train GCN, and obtain a trained model; A first computing unit; the first computing unit is used to calculate an initial path for guiding the UAV to fly based on a standard sawtooth mower strategy at a fixed height above the target area; The second computing unit is used to obtain the current aerial image each time the drone reaches the aerial photography point when flying along the initial path; Perform pixel semantic segmentation on the current aerial image; Path correction unit; The path correction unit takes the waypoint of the drone at the current moment as input and outputs the aerial photography point at the next moment according to the semantic segmentation result; Repeat this process to obtain the corrected drone path.

9. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for adaptive path planning of a drone based on multi-resolution semantic segmentation as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the electronic device, the electronic device executes the method for adaptive path planning of a drone based on multi-resolution semantic segmentation as described in any one of claims 1 to 7.

Images