Unmanned aerial vehicle cruise detection method and device, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of unmanned aerial vehicles, and provides an unmanned aerial vehicle cruise detection method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the global path planning through employing an improved fast extension random tree algorithm; when a conflict risk is detected, a dynamic reward function is designed with returning to a global flight path after obstacle avoidance as a target, and a deep reinforcement learning model is adopted to realize control of actions of the unmanned aerial vehicle; and controlling the unmanned aerial vehicle to fly along the global aircraft path and the local obstacle avoidance path, and executing a target detection task based on the target detection model in the flight process until the task is completed and / or a target position is reached. According to the invention, global path planning is carried out by adopting the improved fast extended random tree algorithm, a local obstacle avoidance path is obtained in combination with the deep reinforcement learning model when facing an emergent obstacle, the autonomous cruising ability of the unmanned aerial vehicle is improved, real-time target detection is realized based on the target detection model in the flight process, and the task execution efficiency of the unmanned aerial vehicle is improved.

Description

Technical Field

[0001] The present application belongs to the field of drone technology, and in particular relates to a drone cruise detection method, device, electronic device, and storage medium. Background Art

[0002] With the rapid development of drone technology, many drone systems have acquired certain autonomous navigation and target detection capabilities. YOLOv7 (an object detection network) is widely used in this field due to its real-time performance and high detection accuracy, enabling real-time obstacle identification. For cruise path planning, the hierarchical planning approach of EGO-planner (a drone path planning algorithm) is widely used to effectively avoid collisions with obstacles.

[0003] However, the existing YOLO-v7 model has certain limitations when handling multi-scale object detection, and traditional path planning algorithms lack performance in obstacle avoidance and energy optimization in dynamic environments. In some complex scenarios, the YOLOv7 model has limited recognition capabilities for objects of uneven scales, resulting in low detection accuracy. Furthermore, traditional path planning algorithms have poor adaptability in dynamic environments, struggling to cope with high-speed obstacles and environmental changes, and require high computational resources, resulting in long planning times. Furthermore, traditional path planning algorithms often overlook energy optimization and have limitations in their ability to optimize multiple objectives, such as endurance, path length, and speed. Summary of the Invention

[0004] The embodiments of the present application provide a drone cruise detection method, device, electronic device, and storage medium method, which can solve the problems of insufficient detection capabilities and limitations in path planning of drones.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting a drone cruise, comprising: Obtain the environmental map data and flight mission data corresponding to the target detection task, use the improved rapidly expanding random tree algorithm to perform global path planning, and obtain the global flight path of the UAV; Detecting a preset area around the drone, and when a collision risk between the drone and an obstacle is detected, using a preset deep reinforcement learning model to control the drone's movements and obtain a local obstacle avoidance path; the deep reinforcement learning model uses current flight state information and obstacle dynamic information as the state space, drone motion instructions as the action space, and designs a dynamic reward function with the goal of returning to the global flight path after avoiding the obstacle; The UAV is controlled to fly along the global aircraft path and the local obstacle avoidance path, and during the flight along the global aircraft path, the target detection task is performed based on a preset target detection model until the target detection task is completed and / or the UAV reaches the target location; wherein the target detection model is trained based on an improved YOLOv7 network model.

[0006] Compared with the prior art, the embodiments of the present application have the following beneficial effects: An improved rapidly expanding random tree algorithm and data corresponding to the target detection task are used for global path planning. Through the path cost optimization mechanism in the algorithm, a global flight path that can simultaneously consider multiple factors is generated. When facing sudden obstacles, the deep reinforcement learning model is combined to adjust the flight action of the UAV. The dynamic reward function is used to carefully evaluate the obstacle avoidance process, so that the UAV returns to the global flight path after avoiding obstacles, thereby obtaining the optimal local obstacle avoidance path and improving the UAV's autonomous cruising capability. In addition, the UAV realizes real-time target detection based on the target detection model during the global flight process, thereby improving the UAV's task execution efficiency.

[0007] In a possible implementation of the first aspect, the steps of obtaining environment map data and flight mission data corresponding to the target detection task, performing global path planning using an improved rapidly expanding random tree algorithm, and obtaining a global flight path of the UAV include: Extracting the flyable area and obstacle information from the environmental map data, and extracting the takeoff position, target position and flight constraints from the flight mission data; Taking the take-off position as the root node of a path tree and the target position as the target node, generating a random sampling point in the flyable area, and finding a neighboring node closest to the random sampling point in the path tree; Generate a new node by extending a first preset step length from the adjacent node to the random sampling point; Checking the path status from the neighboring node to the new node according to the obstacle information, and when an obstacle exists, discarding the new node and returning to the step of generating a random sampling point in the flyable area; when no obstacle exists, adding the new node to the path tree; Find all neighboring nodes in the neighborhood of the new node in the path tree, calculate, for any of the neighboring nodes, a first path cost when the root node is connected to the new node through the neighboring node, and select the neighboring node corresponding to the minimum first path cost as the parent node of the new node; Calculating a second path cost when the root node is connected to the neighboring node through the new node, and when the second path cost is lower than the original path cost of the corresponding neighboring node, replacing the parent node of the neighboring node with the new node; Setting all nodes in the path tree except the root node as leaf nodes, updating the path cost of each leaf node in the path tree, and returning to the step of generating a random sampling point in the flyable area; After each time the new node is added to the path tree, when the distance between the added new node and the target node is less than a preset threshold, tracing back from the target node to the root node according to the path tree to generate an initial path; The initial path is smoothed and adjusted based on the flight constraints to obtain the global flight path.

[0008] In the above scheme, a rough initial path is first generated only by using the environmental constraints in the environmental map data and the improved rapidly expanding random tree algorithm to appropriately reduce the amount of calculation. Then, the initial path is optimized and fine-tuned using the flight constraints in the flight mission data to obtain the optimal global flight path.

[0009] In a possible implementation of the first aspect, the step of updating the path cost of each leaf node in the path tree includes: For any leaf node, extract the path to be updated from the root node to the leaf node; The path cost of the path to be updated is calculated according to the path length and path curvature of the path to be updated and the shortest distance between the path to be updated and an obstacle.

[0010] In the above scheme, by calculating the path cost, multiple objectives can be optimized simultaneously. Among them, the energy consumption of the drone is correlated with the path length and path curvature, and safety is correlated with the shortest distance between the path and obstacles. This allows for planning a global flight path that takes both safety and energy consumption into consideration.

[0011] In a possible implementation of the first aspect, the step of detecting a preset area around the drone includes: Performing obstacle detection in a preset area around the UAV based on the target detection model, and tracking the obstacle when an obstacle is detected; the tracking process includes continuously collecting dynamic obstacle information of the obstacle and continuously obtaining current flight status information of the UAV; The relationship between the distance between the obstacle and the UAV and changes over time is determined based on the current flight state information and the obstacle dynamic information. If the distance between the UAV and the obstacle is always greater than a preset safety obstacle avoidance threshold, tracking of the obstacle is stopped. If the distance between the UAV and the obstacle is less than the preset safety obstacle avoidance threshold at any time, it is determined that there is a risk of collision between the UAV and the obstacle.

[0012] In the above scheme, the target detection model can be used to first identify and detect obstacles, and then the current flight status information of the drone and the dynamic information of the obstacle can be used to further determine whether to perform obstacle avoidance measures. When obstacle avoidance is required, the current flight status information of the drone and the dynamic information of the obstacle can be used as the status basis for the subsequent obstacle avoidance strategy, thereby improving flight efficiency and obstacle avoidance efficiency.

[0013] In a possible implementation of the first aspect, the dynamic reward function is a weighted composite reward function, and the step of designing the dynamic reward function with the goal of returning to the global path after avoiding obstacles includes: Design the first composite reward function with obstacle avoidance as the goal; A second compound reward function is designed with the goal of returning to the global path; wherein the weight distribution of the second compound reward function is different from the weight distribution of the first compound reward function.

[0014] In the above scheme, the use of a weighted composite reward function can optimize multiple objectives simultaneously. In addition, in the multi-stage obstacle avoidance process, by setting a different total target reward for each stage, the weight of each target can be flexibly adjusted to ensure the best overall performance and thus obtain the optimal obstacle avoidance strategy.

[0015] In a possible implementation of the first aspect, the dynamic reward function is expressed as follows:

[0016] in, 、 、 、 、 and are weight coefficients, represents the total reward, Represents the reward for the drone to successfully reach the target location, Indicates the penalty for collision between the drone and obstacles. Represents the progressive reward of the drone from the intermediate target point, Indicates the penalty for the drone staying longer than the preset time. Indicates the reward for the drone to avoid obstacles safely. Represents the reward or penalty for the drone speed control; the intermediate target point is the node position that advances the second preset step length from the current position along the global flight path.

[0017] In a possible implementation of the first aspect, the step of training the target detection model includes: Construct an improved YOLOv7 network model; the improved YOLOv7 network model uses the YOLOv7 network structure as the basic framework, embeds the CBAM module (convolutional block attention module) after the ELAN (efficient layer aggregation network) module in the backbone feature extraction network, replaces the SPPCSPC (spatial pyramid pooling and cross-stage partial connection) module in the neck feature fusion network with the ASPP (atrous spatial convolutional pooling pyramid) module, and replaces the bounding box regression loss function with the SIOU loss function; Obtain a drone aerial photography dataset and convert it to YOLO format, dividing it into a training set and a validation set; The improved YOLOv7 network model is trained and verified using the training set and validation set to obtain a final target detection model.

[0018] In the above scheme, by adding the CBAM module to the YOLOv7 network structure, the model's attention to target features can be enhanced and the accuracy of feature extraction can be improved. The ASPP module is used to replace the original SPPCSPC module to improve the model's ability in multi-scale feature extraction and enhance the detection performance of targets of different sizes. Finally, the SIOU loss function is used to improve the accuracy of bounding box regression, thereby comprehensively improving the target detection model's ability to detect multi-scale targets, effectively completing target detection tasks and assisting path planning.

[0019] In a second aspect, an embodiment of the present application provides a drone cruise detection device, comprising: The global path planning module is used to obtain the environmental map data and flight mission data corresponding to the target detection task, and use the improved rapidly expanding random tree algorithm to perform global path planning to obtain the global flight path of the UAV; A local obstacle avoidance module is configured to detect a preset area around the drone. When a collision risk between the drone and an obstacle is detected, the module uses a preset deep reinforcement learning model to control the drone's movements and obtain a local obstacle avoidance path. The deep reinforcement learning model uses the current flight state information and obstacle dynamic information as the state space, the drone's motion instructions as the action space, and designs a dynamic reward function with the goal of returning to the global flight path after avoiding obstacles. A cruise detection module is configured to control the UAV to fly along the global aircraft path and the local obstacle avoidance path, and to perform the target detection task based on a preset target detection model during flight along the global aircraft path until the target detection task is completed and / or the UAV reaches the target location; wherein the target detection model is trained based on an improved YOLOv7 network model.

[0020] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the drone cruise detection method described in any one of the first aspects above is implemented.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and is characterized in that when the computer program is executed by a processor, it implements the drone cruise detection method described in any one of the first aspects above.

[0022] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the drone cruise detection method described in any one of the first aspects above.

[0023] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 This is a flow chart of a drone cruise detection method provided by an embodiment of the present application; Figure 2 It is a structural diagram of the original YOLOv7 network model of the prior art; Figure 3 1 is a schematic diagram of the structure of an improved YOLOv7 network model provided in another embodiment of the present application; Figure 4 This is a schematic structural diagram of a drone cruise detection device provided in an embodiment of the present application; Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0028] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0030] See also Figure 1 The present application provides a flowchart of a method for detecting a drone cruise. As an example and not a limitation, the method may include the following steps: S11. Obtain environmental map data and flight mission data corresponding to the target detection task, use the improved rapidly expanding random tree algorithm to perform global path planning, and obtain the global flight path of the UAV.

[0031] Among them, the target detection task refers to the positioning and identification of specific targets in a specific area, such as urban road recognition, urban vehicle type detection, and port ship quantity detection.

[0032] S12. Detect the preset area around the drone. When it is detected that there is a risk of conflict between the drone and an obstacle, use the preset deep reinforcement learning model to control the drone's movements and obtain a local obstacle avoidance path.

[0033] Among them, the deep reinforcement learning model uses the current flight state information and obstacle dynamic information as the state space, the drone action instructions as the action space, and designs a dynamic reward function with the goal of returning to the global flight path after avoiding obstacles.

[0034] S13. Control the UAV to fly along the global aircraft path and the local obstacle avoidance path, and perform a target detection task based on a preset target detection model during the flight of the global aircraft path until the target detection task is completed and / or the UAV reaches the target position.

[0035] Among them, the target detection model is trained based on the improved YOLOv7 network model.

[0036] In one possible implementation, the following entire process is started in response to the control instruction of the target detection task. Specifically, the environmental map data and flight mission data corresponding to the target detection task are called to generate a global flight path from the take-off position of the task to the target position. The target detection model and the drone's aircraft are started at the same time. The drone automatically flies according to the global flight path, and the target detection model automatically performs the target detection task. During the flight of the drone, the environment around the drone is synchronously detected. When encountering sudden obstacles, the deep reinforcement learning model is automatically called for local obstacle avoidance, and after avoiding the obstacle, it automatically returns to the global flight path to continue flying and perform the target detection task. Finally, when the drone successfully arrives at the target position or completes the target detection task, the process ends.

[0037] This embodiment provides a drone cruise detection method that uses an improved rapidly expanding random tree algorithm and data corresponding to target detection tasks for global path planning. Through the path cost optimization mechanism in the algorithm, a global flight path that can simultaneously consider multiple factors is generated. When facing sudden obstacles, the drone's flight maneuvers are adjusted in conjunction with a deep reinforcement learning model. The obstacle avoidance process is carefully evaluated using a dynamic reward function, allowing the drone to return to the global flight path after avoiding obstacles, thereby obtaining the optimal local obstacle avoidance path. This improves the drone's autonomous cruise capability, and the drone achieves real-time target detection based on the target detection model during global flight, thereby improving the drone's mission execution efficiency.

[0038] Optionally, an implementation of step S11 may include: S111. Extracting the flyable area and obstacle information from the environmental map data, and extracting the takeoff position, target position, and flight constraints from the flight mission data.

[0039] Among them, the environmental map data includes rough terrain information, building distribution and no-fly zones, etc. Based on the preset flight altitude of the drone, it is detected whether the drone will collide with buildings or slopes, thereby dividing the flyable area and the obstacle area. In one possible implementation method, the no-fly zone can be regarded as a large obstacle.

[0040] In this embodiment, the flight constraints are set based on the requirements of the target detection task and the dynamic characteristics of the drone itself. In one possible implementation, the flight constraints include but are not limited to maximum speed, maximum acceleration, and minimum turning radius.

[0041] In this embodiment, the take-off position and the target position are both three-dimensional positions, usually represented by three coordinate values ​​of X, Y, and Z, which correspond to longitude, latitude, and altitude in geographic space, respectively. It should be noted that the nodes of the subsequent path tree also represent three-dimensional positions.

[0042] S112: Use the take-off position as the root node of the path tree and the target position as the target node, generate a random sampling point in the flyable area, and find the nearest neighboring node to the random sampling point in the path tree.

[0043] S113 , generating a new node by extending a first preset step length from the adjacent node to the random sampling point.

[0044] S114. Check the path status from the adjacent node to the new node according to the obstacle information. If an obstacle exists, discard the new node and return to step S112. If no obstacle exists, add the new node to the path tree.

[0045] S115. Find all neighboring nodes within the neighborhood of the new node in the path tree. For any neighboring node, calculate the first path cost when the root node is connected to the new node through the neighboring nodes, and select the neighboring node corresponding to the minimum first path cost as the parent node of the new node.

[0046] In one possible implementation, a search is performed around the new node within a fixed radius to find the new node's potential parent node set, i.e., all neighboring nodes. The first path cost includes the path cost from the root node to the neighboring node and the neighborhood cost from the neighboring node to the new node.

[0047] S116. Calculate the second path cost when the root node is connected to the neighboring node through the new node. When the second path cost is lower than the original path cost of the corresponding neighboring node, replace the parent node of the neighboring node with the new node.

[0048] The second path cost includes the path cost from the root node to the new node and the path cost from the new node to the neighboring node.

[0049] S117 , setting all nodes except the root node in the path tree as leaf nodes, updating the path cost of each leaf node in the path tree, and returning to step S112 .

[0050] In one possible implementation, the path costs are recalculated only for the leaf nodes affected by steps S115 and S116. For example, if the parent node of the new node is neighboring node A, and the parent node of neighboring node B is changed to the new node, the path costs of neighboring node A, neighboring node B and the new node need to be updated.

[0051] S118. After each execution of step S114, when the distance between the newly added node and the target node is less than a preset threshold, trace back from the target node to the root node according to the path tree to generate an initial path.

[0052] In one possible implementation, in order to avoid falling into an infinite loop, the termination condition of the path planning also includes pre-setting a maximum number of loops, with each execution of step S112 being regarded as a loop, and judging whether the current number of loops has reached the maximum number of loops. If the set maximum number of loops has been reached, it is further detected whether the newly added node is close enough to the target node. If so, it indicates that the path planning is successful and the initial path is output. If not, it indicates that the path planning has failed and a failure message is returned.

[0053] S119: Smoothing the initial path and adjusting the initial path based on flight constraints to obtain a global flight path.

[0054] In one possible implementation, the initial path can be smoothed using methods such as B-spline curves and Bezier curves to reduce the turning points of the path. During the adjustment process, check whether the speed and acceleration of each section on the initial path exceed the maximum speed and maximum acceleration limited by the drone. If they exceed the maximum speed and maximum acceleration, adjust the curvature of the initial path or increase the path length appropriately. Also, check whether the turning radius on the path is not less than the minimum turning radius of the drone. If it is less than the minimum turning radius, adjust the curvature of the path or introduce intermediate nodes so that the initial path meets the flight constraints. Of course, this embodiment is not limited to the above flight constraints. It should be noted that during the fine-tuning process, flight safety is still the main priority to ensure that the path adjustment will not collide with obstacles, and then improve flight efficiency as much as possible to reduce unnecessary flight time and fuel consumption.

[0055] This embodiment provides a drone cruise detection method that first generates a rough initial path using only the environmental constraints in the environmental map data and an improved rapidly expanding random tree algorithm to appropriately reduce the amount of computation. The initial path is then optimized and fine-tuned using the flight constraints in the flight mission data to obtain the optimal global flight path.

[0056] Optionally, an implementation of step S117 includes: S1171. For any leaf node, extract the path to be updated from the root node to the leaf node.

[0057] S1172: Calculate the path cost of the path to be updated based on the path length and path curvature of the path to be updated and the shortest distance between the path to be updated and the obstacle.

[0058] It should be noted that the above implementation is not limited to step S117, but is also applicable when calculating the first path cost and the second path cost in step S115 and step S116.

[0059] In one possible implementation, the path cost The expression is:

[0060] in, 、 and All are weight coefficients; is the path length, the longer the path, the greater the path cost; It represents the reciprocal of the shortest distance between the path and the obstacle. The smaller this distance is, the greater the path cost is. Represents the path curvature. The greater the curvature change, the greater the path cost.

[0061] It's important to note that RTT* (an improved Rapidly Expanding Random Trees algorithm) can generate collision-free paths, which already takes safety into account. Introducing distance from obstacles further enhances safety considerations. Furthermore, the drone's energy consumption is related to path length and path curvature (for example, frequent and sharp turns increase energy consumption), ensuring an energy-optimal solution for the path.

[0062] In this embodiment, the selection of the weight coefficient is based on empirical verification, and depends on both the expert experience in the subjective weighting method and the data analysis results in the objective weighting method.

[0063] This embodiment provides a drone cruise detection method that can simultaneously optimize multiple objectives by calculating path costs. The drone's energy consumption is correlated with path length and path curvature, and safety is correlated with the shortest distance between the path and obstacles. This allows for planning a global flight path that takes both safety and energy consumption into consideration.

[0064] Optionally, an implementation of step S12 specifically includes: S121. Perform obstacle detection in a preset area around the drone based on the target detection model. When an obstacle is detected, track the obstacle.

[0065] Among them, tracking processing includes continuously collecting obstacle dynamic information of obstacles and continuously obtaining the current flight status information of the UAV.

[0066] In one possible implementation, sensors can be placed on the left, right, front, and rear sides of the drone to collect dynamic obstacle information. These sensors can be millimeter-wave radars, which have strong penetration capabilities through fog, smoke, and dust, and offer high ranging accuracy and a long range, effectively ensuring safe flight in complex climates and environments. The drone's current flight status information can be obtained through its own monitoring system or from ground monitoring stations.

[0067] In one possible implementation, in addition to focusing on the positioning and identification of specific targets, the target detection model can also quickly identify obstacles that may affect the flight of the drone. It can also directly monitor the environment in real time through sensors to determine whether there are dynamic obstacles or sudden environmental changes, and then directly collect dynamic information about obstacles around the drone.

[0068] Optionally, the obstacle dynamic information includes the distance between the obstacle and the drone, the obstacle's position relative to the drone, and the obstacle's velocity; the current flight state information includes the drone's position and velocity. Furthermore, the obstacle's motion equations and the drone's motion equations can be derived using the obstacle dynamic information and current flight state information, respectively.

[0069] S122. Determine the relationship between the distance between the obstacle and the UAV over time based on the current flight status information and the obstacle dynamic information. If the distance between the UAV and the obstacle is always greater than a preset safety obstacle avoidance threshold, stop tracking the obstacle. If the distance between the UAV and the obstacle is less than the preset safety obstacle avoidance threshold at any time, determine that there is a risk of collision between the UAV and the obstacle.

[0070] In one possible implementation, the distance between the obstacle and the inspection drone at continuous time points (i.e., moments) is determined based on the motion equations of the inspection drone and the obstacle. The preset safety obstacle avoidance threshold is the minimum collision avoidance distance that should be maintained between the drone and the obstacle, which can be determined based on the drone's flight speed, maneuverability, and environmental factors.

[0071] It should be noted that this embodiment is not limited to the above method to determine whether there is a risk of collision between the drone and the obstacle. When it is detected that the speed direction of the drone relative to the obstacle and the position direction of the obstacle relative to the drone are in the same quadrant, it can be determined that the obstacle will have a certain impact on the flight of the drone. threats.

[0072] The present embodiment provides a method for detecting a drone cruise, which can first use a target detection model to identify and detect obstacles, or use sensors to monitor the environment in real time, and then further determine whether to perform obstacle avoidance measures based on the drone's current flight status information and obstacle dynamic information. When obstacle avoidance is required, the drone's current flight status information and obstacle dynamic information can be used as the status basis for subsequent obstacle avoidance strategies, thereby improving flight efficiency and obstacle avoidance efficiency.

[0073] Optionally, in step S12, the state space of the interaction between the UAV and the environment is constructed using the current flight state information and the obstacle dynamic information. Specifically:

[0074] in, representing three-dimensional position information of an intermediate target point, where the intermediate target point advances from the current position along the global flight path to a node position for a second preset step length. In this embodiment, the second preset step length is set to five unit steps; Indicates the first velocity information, including the speed and acceleration of the drone; Represents the second speed information, including the speed and acceleration of the obstacle; Represents first distance information, that is, the real-time distance between the UAV and the obstacle in each direction. In one possible implementation, the distance between the UAV and the obstacle is obtained in sixteen directions; Represents the second distance information, that is, the deviation distance of the UAV from the global flight path; Indicates the orientation information of the drone, including real-time heading angle and real-time pitch angle.

[0075] Optionally, in step S12, the drone control action is a continuous action space composed of multiple drone action instructions, and the action space of the drone and the environment interaction is constructed through the drone action instructions. Specifically:

[0076] in, Indicates the speed control instructions of the drone, including original speed, acceleration, deceleration and hovering; Indicates the direction control of the drone, including original direction, left turn, right turn, looking up and looking down; Indicates the position control of the drone, including left translation, right translation, translation up, and translation down.

[0077]

[0078] in, 、 、 、 、 and are weight coefficients, represents the total reward, Represents the reward for the drone to successfully reach the target location, Indicates the penalty for collision between the drone and obstacles. Represents the progressive reward of the drone from the intermediate target point, Indicates the penalty for the drone staying longer than the preset time. Indicates the reward or penalty for whether the drone avoids obstacles safely. Indicates the reward or penalty for whether the drone's speed control is within the limit.

[0079] In one possible implementation, the design process of the dynamic reward function includes: S123. Design a first composite reward function with obstacle avoidance as the goal.

[0080] S124. Design a second compound reward function with the goal of returning to the global path; wherein the weight distribution of the second compound reward function is different from the weight distribution of the first compound reward function.

[0081] As an example and not a limitation, when the distance between the drone and the obstacle is lower than the preset distance threshold, the safety obstacle avoidance weight is increased. , reduce the speed weight When the environment is stable and has no obstacles, increase the speed weight To encourage the drone to accelerate its flight and increase the distance weight between the drone and the intermediate target point to encourage drones to stay close to the global flight path, etc.

[0082] For example, when the goal is to avoid obstacles, the first compound reward function is: ; When the goal is to return to the global path, the second compound reward function is: .

[0083] This embodiment provides a drone cruise detection method that uses a weighted composite reward function to simultaneously optimize multiple objectives. In addition, during the multi-stage obstacle avoidance process, by setting a different total target reward for each stage, the weights of each target can be flexibly adjusted to ensure optimal overall performance, thereby obtaining the optimal obstacle avoidance strategy.

[0084] In this embodiment, the UAV needs to detect the entire specific target based on the target detection model during autonomous cruising. Once the detection efficiency or detection accuracy is low, the target detection task will fail. Therefore, the UAV cruise control method also includes improving the target detection model in advance.

[0085] In one possible implementation, the target detection model of this embodiment is trained based on an improved YOLOv7 network model. The training steps of the target detection model include: S21. Build an improved YOLOv7 network model.

[0086] Among them, the improved YOLOv7 network model adopts the YOLOv7 network structure as the basic framework, embeds the CBAM module after the ELAN module in the backbone feature extraction network, replaces the SPPCSPC module in the neck feature fusion network with the ASPP module, and replaces the bounding box regression loss function with the SIOU loss function.

[0087] S22. Obtain the drone aerial photography dataset and convert it into YOLO format, dividing it into training set and validation set.

[0088] S23. Use the training set and validation set to train and validate the improved YOLOv7 network model to obtain the final target detection model.

[0089] As an example and not a limitation, the original YOLOv7 network structure can be referred to Figure 2 As shown, the specific structure of the improved YOLOv7 network model can be referred to Figure 3 shown.

[0090] Specifically, see Figure 2 and Figure 3 , the English meaning and usage of the main modules are: Backbone represents the backbone feature extraction network, which is used to extract multi-scale features of the input image. Its core modules include: CBS module: consists of convolution (Conv), batch normalization (BN) and activation function (such as SiLU); ELAN module: Efficient Layer Aggregation Networks; MP module: downsampling module; CBAM module: convolutional block attention module, such as Figure 3 As shown in the figure, after the CBAM module is added to the ELAN module in a parallel structure, the CBAM module is used to judge the feature attention map in turn from two independent dimensions, namely, spatial and channel, and multiply it with the original feature map for adaptive optimization, thereby strengthening or suppressing related features through learning.

[0091] Neck represents the neck feature fusion network, which is used to fuse the multi-scale features extracted by Backbone to enhance the model's detection capabilities for objects of different sizes. Its core modules include: SPPCSPC module: Spatial Pyramid Pooling -CSPNet (Spatial Pyramid Pooling -CSPNet); UP Sample module: up sampling module; ELAN-W module: A variant of the extended efficient layer aggregation network, which adjusts the branch connection method and convolution kernel combination; ASPP module: Atrous spatial convolution pooling pyramid, such as Figure 3 and Figure 2 As shown in the figure, the ASPP module is used to replace the SPPCSPC module. The ASPP module uses dilated convolutions with different expansion rates to extract features in parallel and generate feature maps with different receptive fields. By sharing the convolution kernel parameters, no additional parameters are needed for multi-scale feature extraction. The output feature maps are fused through channel splicing to retain multi-scale information.

[0092] Head represents the head prediction network, which is used to generate the final detection results on three sets of feature maps of different sizes. Its core modules include: REP module: re-parameterizes convolution, uses a multi-branch structure in the training phase, and fuses it into a single convolution kernel in the inference phase; CBM module: classification and regression module.

[0093] This embodiment provides a method for detecting a drone cruise. By adding a CBAM module to the YOLOv7 network structure, the model's attention to target features can be enhanced, and the accuracy of feature extraction can be improved. The ASPP module is used to replace the original SPPCSPC module to improve the model's ability in multi-scale feature extraction and enhance the detection performance of targets of different sizes. Finally, the SIOU loss function is used to improve the accuracy of bounding box regression, thereby comprehensively improving the target detection model's ability to detect multi-scale targets, effectively completing target detection tasks and assisting in path planning.

[0094] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0095] Corresponding to the drone cruise detection method described in the above embodiment, Figure 4 The structural block diagram of the drone cruise detection device provided in an embodiment of the present application is shown. For the sake of convenience, only the parts related to the embodiment of the present application are shown.

[0096] Reference Figure 4 , the device comprises: The global path planning module 11 is used to obtain the environmental map data and flight mission data corresponding to the target detection task, and use the improved rapidly expanding random tree algorithm to perform global path planning to obtain the global flight path of the UAV.

[0097] The local obstacle avoidance module 12 monitors a predefined area around the drone. If an obstacle is detected that poses a risk of collision with the drone, it uses a predefined deep reinforcement learning model to control the drone's movements and determine a local obstacle avoidance path. This deep reinforcement learning model uses the current flight state and obstacle dynamics as its state space, the drone's motion instructions as its action space, and a dynamic reward function designed with the goal of returning to the global flight path after avoiding the obstacle.

[0098] The cruise detection module 13 is used to control the UAV to fly along the global aircraft path and the local obstacle avoidance path. During flight along the global aircraft path, the module performs target detection tasks based on a preset target detection model until the target detection task is completed and / or the UAV reaches the target location. The target detection model is trained based on a modified YOLOv7 network model.

[0099] In some embodiments of the present application, the global path planning module 11 is specifically used to extract the flyable area and obstacle information from the environmental map data, and to extract the take-off position, target position and flight constraints from the flight mission data; the take-off position is used as the root node of the path tree, the target position is used as the target node, a random sampling point is generated in the flyable area, and the nearest neighboring node to the random sampling point is found in the path tree; a new node is generated by extending a first preset step size from the neighboring node to the random sampling point; the path status from the neighboring node to the new node is checked according to the obstacle information, and when there is an obstacle, the new node is discarded and the step of generating a random sampling point in the flyable area is returned; when there is no obstacle, the new node is added to the path tree; all neighboring nodes within the neighborhood range of the new node are found in the path tree, and for any of the neighboring nodes, a first path cost when the root node is connected to the new node through the neighboring node is calculated, and the neighboring node corresponding to the minimum first path cost is selected as the parent node of the new node; Calculate the second path cost when the root node is connected to the neighboring node through the new node. When there is a second path cost lower than the original path cost of the corresponding neighboring node, replace the parent node of the neighboring node with the new node; set all nodes except the root node in the path tree as leaf nodes, update the path cost of each leaf node in the path tree, and return to the step of generating a random sampling point in the flyable area; after each new node is added to the path tree, when the distance between the added new node and the target node is less than a preset threshold, trace back from the target node to the root node according to the path tree to generate an initial path; smooth the initial path, and adjust the initial path based on the flight constraints to obtain the global flight path.

[0100] In some embodiments of the present application, the global path planning module 11 is also specifically used to extract the path to be updated from the root node to the leaf node for any leaf node; and calculate the path cost of the path to be updated based on the path length, path curvature and the shortest distance between the path to be updated and the obstacle.

[0101] In some embodiments of the present application, the local obstacle avoidance module 12 is specifically used to detect obstacles in a preset area around the drone based on a target detection model, and when an obstacle is detected, the obstacle is tracked; wherein, the tracking process includes continuously collecting obstacle dynamic information of the obstacle and continuously obtaining the current flight status information of the drone; determining the relationship between the distance between the obstacle and the drone over time based on the current flight status information and the obstacle dynamic information; if the distance between the drone and the obstacle is always greater than the preset safety obstacle avoidance threshold, then the tracking of the obstacle is stopped; if the distance between the drone and the obstacle is less than the preset safety obstacle avoidance threshold at any time, then it is determined that there is a risk of conflict between the drone and the obstacle.

[0102] In some embodiments of the present application, the drone cruise detection device also includes a deep reinforcement learning model training module, which is used to design a first compound reward function with obstacle avoidance as the goal; and to design a second compound reward function with returning to the global path as the goal; wherein the weight distribution of the second compound reward function is different from the weight distribution of the first compound reward function.

[0103] In some embodiments of the present application, the drone cruise detection device also includes a target detection model training module for constructing an improved YOLOv7 network model; wherein, the improved YOLOv7 network model adopts the YOLOv7 network structure as the basic framework, embeds the CBAM module after the ELAN module in the backbone feature extraction network, replaces the SPPCSPC module in the neck feature fusion network with the ASPP module, and replaces the bounding box regression loss function with the SIOU loss function; obtains the drone aerial photography data set and converts it into the YOLO format, and divides it into a training set and a validation set; uses the training set and the validation set to train and verify the improved YOLOv7 network model to obtain the final target detection model.

[0104] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0105] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0106] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. Figure 5 As shown, the electronic device 2 of the embodiment includes: at least one processor 20 ( Figure 5 Only one is shown in the figure) a processor, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20. When the processor 20 executes the computer program 22, the steps in the above-mentioned embodiments of the method for detecting abnormalities in each power grid operation and maintenance image are implemented.

[0107] The electronic device 2 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that Figure 5 This is merely an example of the electronic device 2 and does not constitute a limitation on the electronic device 2 . The electronic device 2 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 2 may also include input and output devices, network access devices, etc.

[0108] The processor 20 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0109] In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as the hard drive or memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic device 2, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the electronic device 2. Furthermore, the memory 21 may include both an internal storage unit of the electronic device 2 and an external storage device. The memory 21 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of a computer program. The memory 21 may also be used to temporarily store data that has been output or is about to be output.

[0110] An embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps in each of the above-mentioned UAV cruise detection method embodiments can be implemented.

[0111] An embodiment of the present application provides a computer program product. When the computer program product is run on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned various drone cruise detection method embodiments when executing the computer program product.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, removable hard drives, magnetic disks, or optical disks. In some jurisdictions, based on legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunication signals.

[0113] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0114] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A drone cruise detection method, characterized in that: include: Obtain the environmental map data and flight mission data corresponding to the target detection task, use the improved rapidly expanding random tree algorithm to perform global path planning, and obtain the global flight path of the UAV; Detecting a preset area around the drone, and when a collision risk between the drone and an obstacle is detected, using a preset deep reinforcement learning model to control the drone's movements and obtain a local obstacle avoidance path; the deep reinforcement learning model uses current flight state information and obstacle dynamic information as the state space, drone motion instructions as the action space, and designs a dynamic reward function with the goal of returning to the global flight path after avoiding the obstacle; The UAV is controlled to fly along the global aircraft path and the local obstacle avoidance path, and during the flight along the global aircraft path, the target detection task is performed based on a preset target detection model until the target detection task is completed and / or the UAV reaches the target location; wherein the target detection model is trained based on an improved YOLOv7 network model.

2. The drone cruise detection method according to claim 1, characterized in that: The steps of obtaining the environment map data and flight mission data corresponding to the target detection task, performing global path planning using the improved rapidly expanding random tree algorithm, and obtaining the global flight path of the UAV include: Extracting the flyable area and obstacle information from the environmental map data, and extracting the takeoff position, target position and flight constraints from the flight mission data; Taking the take-off position as the root node of a path tree and the target position as the target node, generating a random sampling point in the flyable area, and finding a neighboring node closest to the random sampling point in the path tree; Generate a new node by extending a first preset step length from the adjacent node to the random sampling point; Checking the path status from the neighboring node to the new node according to the obstacle information, and when an obstacle exists, discarding the new node and returning to the step of generating a random sampling point in the flyable area; when no obstacle exists, adding the new node to the path tree; Find all neighboring nodes in the neighborhood of the new node in the path tree, calculate, for any of the neighboring nodes, a first path cost when the root node is connected to the new node through the neighboring node, and select the neighboring node corresponding to the minimum first path cost as the parent node of the new node; Calculating a second path cost when the root node is connected to the neighboring node through the new node, and when the second path cost is lower than the original path cost of the corresponding neighboring node, replacing the parent node of the neighboring node with the new node; Setting all nodes in the path tree except the root node as leaf nodes, updating the path cost of each leaf node in the path tree, and returning to the step of generating a random sampling point in the flyable area; After each time the new node is added to the path tree, when the distance between the added new node and the target node is less than a preset threshold, tracing back from the target node to the root node according to the path tree to generate an initial path; The initial path is smoothed and adjusted based on the flight constraints to obtain the global flight path.

3. The drone cruise detection method according to claim 2, characterized in that: The step of updating the path cost of each leaf node in the path tree comprises: For any leaf node, extract the path to be updated from the root node to the leaf node; The path cost of the path to be updated is calculated according to the path length and path curvature of the path to be updated and the shortest distance between the path to be updated and an obstacle.

4. The drone cruise detection method according to claim 1, characterized in that: The step of detecting a preset area around the drone includes: Performing obstacle detection in a preset area around the UAV based on the target detection model, and tracking the obstacle when an obstacle is detected; the tracking process includes continuously collecting dynamic obstacle information of the obstacle and continuously obtaining current flight status information of the UAV; The relationship between the distance between the obstacle and the UAV and changes over time is determined based on the current flight state information and the obstacle dynamic information. If the distance between the UAV and the obstacle is always greater than a preset safety obstacle avoidance threshold, tracking of the obstacle is stopped. If the distance between the UAV and the obstacle is less than the preset safety obstacle avoidance threshold at any time, it is determined that there is a risk of collision between the UAV and the obstacle.

5. The drone cruise detection method according to claim 1, characterized in that: The dynamic reward function is a weighted composite reward function, and the step of designing the dynamic reward function with the goal of returning to the global path after avoiding obstacles includes: Design the first composite reward function with obstacle avoidance as the goal; A second compound reward function is designed with the goal of returning to the global path; wherein the weight distribution of the second compound reward function is different from the weight distribution of the first compound reward function.

6. The drone cruise detection method according to claim 5, characterized in that: The expression of the dynamic reward function is: ； in, 、 、 、 、 and are weight coefficients, represents the total reward, Represents the reward for the drone to successfully reach the target location, Indicates the penalty for collision between the drone and obstacles. Represents the progressive reward of the drone from the intermediate target point, Indicates the penalty for the drone staying longer than the preset time. Indicates the reward for the drone to avoid obstacles safely. Represents the reward or penalty for the drone speed control; the intermediate target point is the node position that advances the second preset step length from the current position along the global flight path.

7. The drone cruise detection method according to claim 1, characterized in that: The training steps of the target detection model include: Construct an improved YOLOv7 network model; the improved YOLOv7 network model adopts the YOLOv7 network structure as the basic framework, embeds the CBAM module after the ELAN module in the backbone feature extraction network, replaces the SPPCSPC module in the neck feature fusion network with the ASPP module, and replaces the bounding box regression loss function with the SIOU loss function; Obtain a drone aerial photography dataset and convert it to YOLO format, dividing it into a training set and a validation set; The improved YOLOv7 network model is trained and verified using the training set and validation set to obtain a final target detection model.

8. A drone cruise detection device, characterized in that: include: The global path planning module is used to obtain the environmental map data and flight mission data corresponding to the target detection task, and use the improved rapidly expanding random tree algorithm to perform global path planning to obtain the global flight path of the UAV; A local obstacle avoidance module is configured to detect a preset area around the drone. When a collision risk between the drone and an obstacle is detected, the module uses a preset deep reinforcement learning model to control the drone's movements and obtain a local obstacle avoidance path. The deep reinforcement learning model uses the current flight state information and obstacle dynamic information as the state space, the drone's motion instructions as the action space, and designs a dynamic reward function with the goal of returning to the global flight path after avoiding obstacles. A cruise detection module is configured to control the UAV to fly along the global aircraft path and the local obstacle avoidance path, and to perform the target detection task based on a preset target detection model during flight along the global aircraft path until the target detection task is completed and / or the UAV reaches the target location; wherein the target detection model is trained based on an improved YOLOv7 network model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the drone cruise detection method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the drone cruise detection method according to any one of claims 1 to 7 is implemented.

Images