Methods, devices, electronic equipment and storage media for drone cruise detection

Abstract

This application relates to the field of unmanned aerial vehicle (UAV) technology, and provides a UAV cruise detection method, device, electronic device, and storage medium. The method includes: performing global path planning using an improved fast expanding random tree algorithm; when a conflict risk is detected, designing a dynamic reward function with the goal of returning to the global flight path after obstacle avoidance, and using a deep reinforcement learning model to control the UAV's actions; controlling the UAV to fly along the global flight path and local obstacle avoidance paths, and performing target detection tasks based on a target detection model during flight until the task is completed and / or the target position is reached. This application uses an improved fast expanding random tree algorithm for global path planning, and when facing sudden obstacles, combines a deep reinforcement learning model to obtain local obstacle avoidance paths, improving the UAV's autonomous cruise capability. Furthermore, it achieves real-time target detection based on a target detection model during flight, improving the UAV's task execution efficiency.

Description

Technical Field

[0001] This application belongs to the field of unmanned aerial vehicle (UAV) technology, and particularly relates to a UAV cruise detection method, device, electronic equipment, and storage medium. Background Technology

[0002] With the rapid development of drone technology, many drone systems have acquired certain autonomous navigation and target detection capabilities. Among them, YOLOv7 (a target detection network) is widely used in the field of target detection due to its real-time performance and high detection accuracy, and can identify obstacles in the environment in real time. In terms of cruise path planning, the hierarchical planning method of EGO-planner (a drone path planning algorithm) is widely used, which can effectively avoid collisions with obstacles.

[0003] However, the existing YOLO-v7 model has certain limitations in handling multi-scale object detection, and traditional path planning algorithms are inadequate in obstacle avoidance and energy consumption optimization in dynamic environments. In some complex scenarios, the YOLOv7 model has limited ability to identify targets with uneven scales, resulting in low detection accuracy. At the same time, traditional path planning algorithms have poor adaptability in dynamic environments, making it difficult to cope with high-speed moving obstacles and environmental changes, and they have high computational resource requirements, resulting in long planning times. In addition, traditional path planning algorithms often neglect energy consumption optimization and have limitations in optimizing various objectives such as endurance, path length, and speed. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for detecting unmanned aerial vehicle (UAV) cruises, which can solve the problems of insufficient detection capabilities and limitations in path planning of UAVs.

[0005] In a first aspect, embodiments of this application provide a method for detecting unmanned aerial vehicle (UAV) cruise, including: The environmental map data and flight mission data corresponding to the target detection task are acquired, and the global path planning is performed using the improved fast expanding random tree algorithm to obtain the global flight path of the UAV. The system detects a preset area around the drone. When an obstacle is detected that poses a risk of conflict with the drone, a preset deep reinforcement learning model is used to control the drone's actions and obtain a local obstacle avoidance path. The deep reinforcement learning model uses the current flight state information and obstacle dynamic information as the state space, the drone's action commands as the action space, and the goal of returning to the global flight path after obstacle avoidance to design a dynamic reward function. The UAV is controlled to fly along the global flight path and the local obstacle avoidance path, and during the flight along the global flight path, the UAV performs the target detection task based on a preset target detection model until the target detection task is completed and / or the UAV reaches the target position; wherein, the target detection model is trained based on an improved YOLOv7 network model.

[0006] The beneficial effects of the embodiments in this application compared with the prior art are: An improved fast expanding random tree algorithm and data corresponding to the target detection task are used for global path planning. Through the path cost optimization mechanism in the algorithm, a global flight path that can consider multiple factors at the same time is generated. When facing sudden obstacles, the UAV's flight behavior is adjusted by combining a deep reinforcement learning model. The obstacle avoidance process is evaluated in detail using a dynamic reward function, so that the UAV returns to the global flight path after obstacle avoidance, thereby obtaining the optimal local obstacle avoidance path, improving the UAV's autonomous cruise capability. Furthermore, the UAV achieves real-time target detection based on the target detection model during global flight, improving the UAV's mission execution efficiency.

[0007] In one possible implementation of the first aspect, the steps of acquiring environmental map data and flight mission data corresponding to the target detection task, and using an improved fast extended random tree algorithm for global path planning to obtain the global flight path of the UAV include: The flightable area and obstacle information are extracted from the environmental map data, and the takeoff position, target position and flight constraints are extracted from the flight mission data. Using the takeoff position as the root node of the path tree and the target position as the target node, a random sampling point is generated in the flyable area, and the nearest neighbor node to the random sampling point is found in the path tree; New nodes are generated by extending the random sampling points from the neighboring nodes by a first preset step size. Based on the obstacle information, check the path status from the neighboring nodes to the new node. If an obstacle exists, discard the new node and return to the step of generating a random sampling point in the flyable area; if no obstacle exists, add the new node to the path tree. Find all neighboring nodes within the neighborhood range of the new node in the path tree. For any one of the neighboring nodes, calculate the first path cost when the root node is connected to the new node through the neighboring node. Select the neighboring node with the minimum first path cost as the parent node of the new node. Calculate the second path cost when the root node is connected to the neighboring node through the new node. If the second path cost is lower than the original path cost of the corresponding neighboring node, replace the parent node of the neighboring node with the new node. Set all nodes in the path tree except the root node as leaf nodes, update the path cost of each leaf node in the path tree, and return to the step of generating a random sampling point in the flyable area; After each new node is added to the path tree, if the distance between the added new node and the target node is less than a preset threshold, an initial path is generated by backtracking from the target node to the root node according to the path tree. The initial path is smoothed and adjusted based on the flight constraints to obtain the global flight path.

[0008] In the above scheme, a rough initial path is first generated using only environmental constraints in the environmental map data and an improved fast expanding random tree algorithm to appropriately reduce the amount of computation. Then, the initial path is optimized and fine-tuned using flight constraints in the flight mission data to obtain the optimal global flight path.

[0009] In one possible implementation of the first aspect, the step of updating the path cost of each leaf node in the path tree includes: For any leaf node, extract the path to be updated from the root node to the leaf node; The path cost of the path to be updated is calculated based on the path length, path curvature, and the shortest distance between the path to be updated and the obstacle.

[0010] In the above scheme, by calculating the path cost, multiple objectives can be optimized simultaneously. Among them, the energy consumption of the UAV is related to the path length and path curvature, and the safety is related to the shortest distance between the path and obstacles. Thus, a global flight path that takes into account both safety and energy consumption can be planned.

[0011] In one possible implementation of the first aspect, the step of detecting a predetermined area around the drone includes: Based on the target detection model, obstacles are detected in a preset area around the UAV. When an obstacle is detected, the obstacle is tracked. The tracking process includes continuously collecting the obstacle's dynamic information and continuously acquiring the UAV's current flight status information. Based on the current flight status information and the obstacle dynamic information, the relationship between the distance between the obstacle and the drone over time is determined. If the distance between the drone and the obstacle is always greater than a preset safe obstacle avoidance threshold, the tracking of the obstacle is stopped. If the distance between the drone and the obstacle is less than the preset safe obstacle avoidance threshold at any time, it is determined that there is a risk of conflict between the drone and the obstacle.

[0012] In the above scheme, obstacles can be identified and detected first using a target detection model. Then, the current flight status information of the UAV and the dynamic information of the obstacles can be used to further determine whether to implement obstacle avoidance measures. When obstacle avoidance is required, the current flight status information of the UAV and the dynamic information of the obstacles can be used as the status basis for subsequent obstacle avoidance strategies, thereby improving flight efficiency and obstacle avoidance efficiency.

[0013] In one possible implementation of the first aspect, the dynamic reward function is a weighted composite reward function, and the step of designing the dynamic reward function with the goal of returning to the global path after obstacle avoidance includes: The first composite reward function is designed with obstacle avoidance as the objective. A second composite reward function is designed with the goal of returning the global path; wherein the weight allocation of the second composite reward function is different from that of the first composite reward function.

[0014] In the above scheme, the use of a weighted composite reward function can optimize multiple objectives simultaneously. In addition, in the multi-stage obstacle avoidance process, by setting different total objective rewards for each stage, the weights of each objective can be flexibly adjusted to ensure optimal overall performance, thereby obtaining the optimal obstacle avoidance strategy.

[0015] In one possible implementation of the first aspect, the expression for the dynamic reward function is:

[0016] in, , , , , and All are weighting coefficients. Indicates the total reward. The reward indicates that the drone has successfully reached the target location. This indicates the penalty for a drone colliding with an obstacle. This represents the progressive reward for the drone's distance from the intermediate target point. This indicates the penalty for the drone staying for an extended period of time. This indicates a reward for the drone's safe obstacle avoidance. This indicates a reward or penalty for drone speed control; the intermediate target point is a node position where the drone advances a second preset step length along the global flight path from its current position.

[0017] In one possible implementation of the first aspect, the training steps of the object detection model include: An improved YOLOv7 network model is constructed. The improved YOLOv7 network model adopts the YOLOv7 network structure as the basic framework. The CBAM module (convolutional block attention module) is embedded after the ELAN (efficient layer aggregation network) module in the backbone feature extraction network. The SPPCSPC (spatial pyramid pooling and cross-stage partial connectivity) module in the neck feature fusion network is replaced with the ASPP (dilated spatial convolutional pooling pyramid) module, and the bounding box regression loss function is changed to the SIOU loss function. Acquire the drone aerial photography dataset and convert it to YOLO format, then divide it into training and validation sets; The improved YOLOv7 network model is trained and validated using the training and validation sets to obtain the final object detection model.

[0018] In the above scheme, adding the CBAM module to the YOLOv7 network structure can enhance the model's attention to target features and improve the accuracy of feature extraction. Replacing the original SPPC module with the ASPP module can improve the model's ability to extract features at multiple scales and enhance the detection performance of targets of different sizes. Finally, the SIOU loss function is used to improve the accuracy of bounding box regression, thereby comprehensively improving the target detection model's ability to detect targets at multiple scales, effectively completing the target detection task and assisting in path planning.

[0019] Secondly, embodiments of this application provide a drone cruise detection device, comprising: The global path planning module is used to acquire environmental map data and flight mission data corresponding to the target detection task, and to perform global path planning using an improved fast expanding random tree algorithm to obtain the global flight path of the UAV. The local obstacle avoidance module is used to detect a preset area around the UAV. When an obstacle is detected that poses a risk of conflict with the UAV, a preset deep reinforcement learning model is used to control the UAV's actions and obtain a local obstacle avoidance path. The deep reinforcement learning model uses the current flight state information and obstacle dynamic information as the state space, the UAV action commands as the action space, and the goal of returning to the global flight path after obstacle avoidance to design a dynamic reward function. The cruise detection module is used to control the UAV to fly along the global aircraft path and the local obstacle avoidance path, and to perform the target detection task based on a preset target detection model during the flight along the global aircraft path, until the target detection task is completed and / or the UAV reaches the target position; wherein, the target detection model is trained based on an improved YOLOv7 network model.

[0020] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the UAV cruise detection method described in any one of the first aspects above.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the UAV cruise detection method described in any one of the first aspects above.

[0022] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the unmanned aerial vehicle (UAV) cruise detection method described in any one of the first aspects.

[0023] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic flowchart of a drone cruise detection method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the original YOLOv7 network model in existing technology; Figure 3 This is a schematic diagram of the structure of an improved YOLOv7 network model provided in another embodiment of this application; Figure 4 This is a schematic diagram of the structure of the UAV cruise detection device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0028] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] See Figure 1 This application provides a flowchart of a drone cruise detection method, which is provided as an example and not a limitation. The method may include the following steps: S11. Obtain environmental map data and flight mission data corresponding to the target detection task, and use the improved fast extended random tree algorithm to perform global path planning to obtain the global flight path of the UAV.

[0031] Among them, target detection tasks refer to the location and identification of specific targets within a specific area, such as urban road identification, urban vehicle type detection, and port ship quantity detection.

[0032] S12. Detect the preset area around the drone. When an obstacle is detected that poses a risk of conflict with the drone, use a preset deep reinforcement learning model to control the drone's actions and obtain a local obstacle avoidance path.

[0033] The deep reinforcement learning model uses the current flight state information and obstacle dynamic information as the state space, the UAV action commands as the action space, and the goal of returning to the global flight path after obstacle avoidance as the objective to design a dynamic reward function.

[0034] S13. Control the UAV to fly along the global aircraft path and the local obstacle avoidance path, and perform target detection tasks based on the preset target detection model during the flight of the global aircraft path, until the target detection task is completed and / or the UAV reaches the target position.

[0035] The object detection model was trained based on an improved YOLOv7 network model.

[0036] In one possible implementation, the following process begins in response to the control command for the target detection task: Specifically, the environmental map data and flight task data corresponding to the target detection task are called to generate a global flight path from the takeoff position to the target position. The target detection model and the UAV are started simultaneously. The UAV automatically flies according to the global flight path, while the target detection model automatically executes the target detection task. During the flight of the UAV, the surrounding environment is detected synchronously. When encountering a sudden obstacle, the deep reinforcement learning model is automatically called to perform local obstacle avoidance. After obstacle avoidance, the UAV automatically returns to the global flight path to continue flying and executing the target detection task. Finally, the process ends when the UAV successfully reaches the target position or completes the target detection task.

[0037] This embodiment provides a UAV cruise detection method that uses an improved fast extended random tree algorithm and data corresponding to the target detection task for global path planning. Through the path cost optimization mechanism in the algorithm, a global flight path that can consider multiple factors simultaneously is generated. When facing sudden obstacles, the flight behavior of the UAV is adjusted by combining a deep reinforcement learning model. The obstacle avoidance process is carefully evaluated using a dynamic reward function, so that the UAV returns to the global flight path after obstacle avoidance, thereby obtaining the optimal local obstacle avoidance path, improving the autonomous cruise capability of the UAV. Furthermore, the UAV achieves real-time target detection based on the target detection model during global flight, improving the task execution efficiency of the UAV.

[0038] Optionally, one implementation of step S11 may include: S111. Extract information on flyable areas and obstacles from environmental map data, and extract takeoff position, target position, and flight constraints from flight mission data.

[0039] The environmental map data includes rough terrain information, building distribution, and no-fly zones. Based on the drone's preset flight altitude, it detects whether the drone will collide with buildings or slopes, thereby dividing the flyable area and the area where obstacles are located. In one possible implementation, the no-fly zone can be regarded as a large obstacle.

[0040] In this embodiment, the flight constraints are set based on the requirements of the target detection task and the dynamic characteristics of the UAV itself. In one possible implementation, the flight constraints include, but are not limited to, maximum speed, maximum acceleration, and minimum turning radius.

[0041] In this embodiment, both the takeoff position and the target position are three-dimensional positions, usually represented by three coordinate values: X, Y, and Z, which correspond to longitude, latitude, and altitude in geographic space, respectively. It should be noted that the nodes of the subsequent path tree also represent three-dimensional positions.

[0042] S112. Using the takeoff position as the root node of the path tree and the target position as the target node, generate a random sampling point in the flyable area and find the nearest neighbor node to the random sampling point in the path tree.

[0043] S113. Generate new nodes by expanding from neighboring nodes to random sampling points by a first preset step size.

[0044] S114. Check the path status from neighboring nodes to the new node based on obstacle information. If there is an obstacle, discard the new node and return to step S112; if there is no obstacle, add the new node to the path tree.

[0045] S115. Find all neighboring nodes within the neighborhood of the new node in the path tree. For any neighboring node, calculate the first path cost when the root node is connected to the new node through the neighboring node. Select the neighboring node with the minimum first path cost as the parent node of the new node.

[0046] In one possible implementation, a search is conducted around the new node with a fixed radius, targeting the set of potential parent nodes of the new node, i.e., all neighboring nodes. The first path cost includes the path cost from the root node to a neighboring node and the neighborhood cost from a neighboring node to the new node.

[0047] S116. Calculate the second path cost when the root node is connected to a neighboring node through the new node. If the second path cost is lower than the original path cost of the corresponding neighboring node, replace the parent node of the neighboring node with the new node.

[0048] The second path cost includes the path cost from the root node to the new node and the path cost from the new node to the neighboring nodes.

[0049] S117. Set all nodes in the path tree except the root node as leaf nodes, update the path cost of each leaf node in the path tree, and return to step S112.

[0050] In one possible implementation, the path cost is recalculated only for the leaf nodes affected by steps S115 and S116. For example, if the parent node of the new node is neighbor node A, and the parent node of neighbor node B is changed to the new node, then the path costs of neighbor node A, neighbor node B, and the new node need to be updated.

[0051] S118. After each execution of step S114, when the distance between the newly added node and the target node is less than a preset threshold, the initial path is generated by backtracking from the target node to the root node according to the path tree.

[0052] In one possible implementation, to avoid getting stuck in an infinite loop, the termination condition for path planning also includes setting a maximum number of loops in advance, taking each execution of step S112 as one loop, and determining whether the current number of loops has reached the maximum number of loops. If the maximum number of loops has been reached, it further checks whether the newly added node is close enough to the target node. If so, it means that the path planning is successful and the initial path is output; otherwise, it means that the path planning has failed and a failure message is returned.

[0053] S119. Smooth the initial path and adjust it based on flight constraints to obtain the global flight path.

[0054] In one possible implementation, B-spline curves, Bézier curves, or similar methods can be used to smooth the initial path, reducing inflection points. During adjustment, it is checked whether the velocity and acceleration of each segment on the initial path exceed the maximum speed and acceleration limits of the UAV. If they do, the curvature of the initial path is adjusted or the path length is appropriately increased. Simultaneously, it is checked whether the turning radius on the path is not less than the minimum turning radius of the UAV. If it is less than the minimum turning radius, the curvature of the path is adjusted or intermediate nodes are introduced to ensure the initial path meets flight constraints. Of course, this embodiment is not limited to the above flight constraints. It should be noted that during fine-tuning, flight safety remains the primary focus, ensuring that the path adjustment does not collide with obstacles, while maximizing flight efficiency and minimizing unnecessary flight time and fuel consumption.

[0055] This embodiment provides a UAV cruise detection method that first generates a rough initial path using only environmental constraints in the environmental map data and an improved fast expanding random tree algorithm to appropriately reduce the amount of computation. Then, the initial path is optimized and fine-tuned using flight constraints in the flight mission data to obtain the optimal global flight path.

[0056] Optionally, one implementation of step S117 includes: S1171. For any leaf node, extract the path to be updated from the root node to the leaf node.

[0057] S1172. Calculate the path cost of the path to be updated based on the path length, path curvature, and the shortest distance between the path to be updated and the obstacle.

[0058] It should be noted that the above implementation method is not limited to step S117, but also applies to calculating the first path cost and the second path cost in steps S115 and S116.

[0059] In one possible implementation, path cost The expression is:

[0060] in, , and All are weighting coefficients; The longer the path, the greater the path cost. This represents the reciprocal of the shortest distance between the path and the obstacle; the smaller this distance, the greater the path cost. The curvature of the path is represented by the curvature. The greater the change in curvature, the higher the path cost.

[0061] It should be noted that while RTT* (Improved Fast Expanding Random Tree algorithm) can generate collision-free paths, which already takes safety into account, introducing the distance cost to obstacles further addresses safety concerns. Simultaneously, the drone's energy consumption is related to path length and path curvature (e.g., frequent sharp turns consume more energy), ensuring the optimal energy consumption solution for the path is achieved.

[0062] In this embodiment, the selection of weighting coefficients is based on empirical verification, relying on both expert experience in the subjective weighting method and data analysis results in the objective weighting method.

[0063] This embodiment provides a drone cruise detection method that can simultaneously optimize multiple objectives by calculating path costs. The drone's energy consumption is related to path length and path curvature, while its safety is related to the shortest distance between the path and obstacles. This allows for the planning of a global flight path that considers both safety and energy consumption.

[0064] Optionally, one implementation of step S12 specifically includes: S121. Based on the target detection model, perform obstacle detection in the preset area around the UAV. When an obstacle is detected, perform tracking processing on the obstacle.

[0065] The tracking process includes continuously collecting obstacle dynamics information and continuously acquiring the current flight status information of the UAV.

[0066] In one possible implementation, sensors can be positioned on the left, right, front, and rear sides of the drone to collect dynamic information about obstacles. These sensors can be millimeter-wave radars, whose sensors have strong penetrating capabilities through fog, smoke, and dust, offering high ranging accuracy and a long range, thus providing strong support for the drone's safe flight in complex weather and environmental conditions. The drone's current flight status information can be obtained through the drone's own monitoring system or through a ground monitoring station.

[0067] In one possible implementation, in addition to focusing on the localization and identification of specific targets, the target detection model can also quickly identify obstacles that may affect the flight of the drone. It can also directly monitor the environment in real time through sensors to determine whether there are dynamic obstacles or sudden environmental changes, and then directly collect dynamic information of obstacles around the drone.

[0068] Optionally, obstacle dynamic information includes the distance between the obstacle and the UAV, the obstacle's position relative to the UAV, and the obstacle's speed; current flight status information includes the UAV's position and speed. Furthermore, the obstacle's motion equations and the UAV's motion equations can be obtained from the obstacle dynamic information and the current flight status information, respectively.

[0069] S122. Determine the relationship between the distance between the drone and the obstacle over time based on the current flight status information and obstacle dynamic information. If the distance between the drone and the obstacle is always greater than the preset safe obstacle avoidance threshold, stop tracking the obstacle. If the distance between the drone and the obstacle is less than the preset safe obstacle avoidance threshold at any time, determine that there is a risk of conflict between the drone and the obstacle.

[0070] In one possible implementation, the distance between the obstacle and the inspection drone at consecutive time points (i.e., moments) is determined based on the motion equations of the inspection drone and the obstacle. The preset safe obstacle avoidance threshold is the minimum collision avoidance distance that the drone and the obstacle should maintain, which can be determined based on the drone's flight speed, maneuverability, and environmental factors.

[0071] It should be noted that this embodiment is not limited to the above-described method for determining whether there is a risk of conflict between the drone and an obstacle. It can also determine that the obstacle will affect the drone's flight when the drone's velocity direction relative to the obstacle and the obstacle's position direction relative to the drone are in the same quadrant. The threat.

[0072] This embodiment provides a drone cruise detection method that can first use a target detection model to identify and detect obstacles, or use sensors to monitor the environment in real time. Then, it can further determine whether to implement obstacle avoidance measures by using the drone's current flight status information and obstacle dynamic information. When obstacle avoidance is required, the drone's current flight status information and obstacle dynamic information can be used as the status basis for subsequent obstacle avoidance strategies, thereby improving flight efficiency and obstacle avoidance efficiency.

[0073] Optionally, in step S12, a state space for the interaction between the UAV and the environment is constructed using the current flight state information and obstacle dynamic information, specifically:

[0074] in, The three-dimensional position information of the intermediate target point is represented by the node position where the intermediate target point moves forward a second preset step length along the global flight path from its current position. In this embodiment, the second preset step length is set to five unit steps. This indicates the initial velocity information, including the drone's speed and acceleration; This indicates the second velocity information, including the speed and acceleration of the obstacle; This represents the first distance information, namely the real-time distance between the drone and the obstacle in each direction. In one possible implementation, the distance between the drone and the obstacle is obtained in sixteen directions. This represents the second distance information, namely the deviation of the drone from the global flight path; This indicates the drone's orientation information, including real-time heading angle and real-time pitch angle.

[0075] Optionally, in step S12, the UAV control action is a continuous action space composed of multiple UAV action commands. The action space for interaction between the UAV and the environment is constructed through the UAV action commands. Specifically:

[0076] in, Indicates the speed control commands for the drone, including normal speed, acceleration, deceleration, and hovering; This indicates the drone's orientation control, including its original direction, left turn, right turn, looking up, and looking down; This refers to the position control of the drone, including left translation, right translation, translation up, and translation down.

[0077]

[0078] in, , , , , and All are weighting coefficients. Indicates the total reward. The reward indicates that the drone has successfully reached the target location. This indicates the penalty for a drone colliding with an obstacle. This represents the progressive reward for the drone's distance from the intermediate target point. This indicates the penalty for the drone staying for an extended period of time. Indicating rewards or penalties for whether a drone safely avoids obstacles. This indicates whether the drone's speed control is within the specified limits, and whether there are rewards or penalties.

[0079] In one possible implementation, the design process of the dynamic reward function includes: S123. Design the first composite reward function with obstacle avoidance as the goal.

[0080] S124. Design a second composite reward function with the goal of returning to the global path; wherein the weight allocation of the second composite reward function is different from that of the first composite reward function.

[0081] As an example, and not a limitation, when the distance between the drone and the obstacle is below a preset distance threshold, the safety obstacle avoidance weight is increased. Reduce speed weight And when the environment is stable and unobstructed, increase the speed weight. This encourages drones to accelerate and increases the weighting of distance between the drone and the intermediate target point. This encourages drones to approach the global flight path, etc.

[0082] For example, when the goal is obstacle avoidance, the first composite reward function is: ; When the goal is to return to the global path, the second composite reward function is: .

[0083] This embodiment provides a drone cruise detection method that uses a weighted composite reward function to optimize multiple targets simultaneously. In addition, during the multi-stage obstacle avoidance process, by setting different total target rewards for each stage, the weights of each target are flexibly adjusted to ensure optimal overall performance and thus obtain the optimal obstacle avoidance strategy.

[0084] In this embodiment, the UAV needs to detect the entire specific target based on the target detection model during autonomous cruise. If the detection efficiency is low or the detection accuracy is low, the target detection task will fail. Therefore, the UAV cruise control method also includes improving the target detection model in advance.

[0085] In one possible implementation, the object detection model in this embodiment is trained based on an improved YOLOv7 network model. The training steps of this object detection model include: S21. Construct an improved YOLOv7 network model.

[0086] The improved YOLOv7 network model uses the YOLOv7 network structure as its basic framework. It embeds the CBAM module after the ELAN module in the backbone feature extraction network, replaces the SPPCSPC module in the neck feature fusion network with the ASPP module, and changes the bounding box regression loss function to the SIOU loss function.

[0087] S22. Obtain the drone aerial photography dataset and convert it to YOLO format, then divide it into training and validation sets.

[0088] S23. The improved YOLOv7 network model is trained and validated using the training and validation sets to obtain the final object detection model.

[0089] As an example rather than a limitation, the original YOLOv7 network structure can be referenced. Figure 2 As shown, the specific structure of the improved YOLOv7 network model can be found in [reference needed]. Figure 3 As shown.

[0090] Specifically, see Figure 2 and Figure 3 The English meanings and usages of the main modules are as follows: Backbone refers to the backbone feature extraction network, which is used to extract multi-scale features from the input image. Its core modules include: The CBS module consists of convolution (Conv), batch normalization (BN), and activation functions (such as SiLU). ELAN module: Efficient Layer Aggregation Networks; MP module: Downsampling module; CBAM module: Convolutional Block Attention Module, such as... Figure 3 As shown, after the CBAM module is added to the ELAN module in a parallel structure, the CBAM module is used to judge the feature attention map from two independent dimensions, space and channel, respectively, and multiply it with the original feature map for adaptive optimization, thereby learning to strengthen or suppress related features.

[0091] Neck refers to the neck feature fusion network, which integrates multi-scale features extracted from the backbone to enhance the model's ability to detect targets of different sizes. Its core modules include: SPPC module: Spatial Pyramid Pooling - Cross-Stage Partial Network (CSPNet); UP Sample module: Upsampling module; ELAN-W module: A variant of extended high-efficiency layer aggregation networks, which adjusts the connection of branches and the combination of convolution kernels; ASPP module: dilated spatial convolutional pooling pyramid, such as Figure 3 and Figure 2 As shown, the ASPP module replaces the SPPCSPC module. The ASPP module uses dilated convolutions with different dilation rates to extract features in parallel, generating feature maps with different receptive fields. By sharing the convolution kernel parameters, no additional parameters need to be added when extracting features at multiple scales. The output feature maps are fused by channel concatenation to retain multi-scale information.

[0092] Head refers to the head prediction network, which generates the final detection result on three sets of feature maps of different sizes. Its core modules include: REP module: Reparameterized convolution, which uses a multi-branch structure during the training phase and merges it into a single convolution kernel during the inference phase; CBM module: Classification and Regression module.

[0093] This embodiment provides a drone cruise detection method. By adding a CBAM module to the YOLOv7 network structure, the model's attention to target features can be enhanced, improving the accuracy of feature extraction. Replacing the original SPPCSPC module with an ASPP module can improve the model's ability to extract features at multiple scales, enhancing the detection performance of targets of different sizes. Finally, the SIOU loss function is used to improve the accuracy of bounding box regression, thereby comprehensively improving the target detection model's ability to detect targets at multiple scales. This method can effectively complete target detection tasks and assist in path planning.

[0094] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0095] Corresponding to the UAV cruise detection method described in the above embodiments, Figure 4 The diagram shows a structural block diagram of the UAV cruise detection device provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0096] Reference Figure 4 The device includes: The global path planning module 11 is used to acquire environmental map data and flight mission data corresponding to the target detection task, and to perform global path planning using an improved fast expanding random tree algorithm to obtain the global flight path of the UAV.

[0097] The local obstacle avoidance module 12 is used to detect a preset area around the drone. When an obstacle is detected that poses a risk of conflict with the drone, a preset deep reinforcement learning model is used to control the drone's actions and obtain a local obstacle avoidance path. The deep reinforcement learning model uses the current flight state information and obstacle dynamic information as the state space, the drone's action commands as the action space, and a dynamic reward function designed with the goal of returning to the global flight path after obstacle avoidance.

[0098] The cruise detection module 13 is used to control the UAV to fly along the global flight path and the local obstacle avoidance path, and to perform target detection tasks based on a preset target detection model during the flight along the global flight path, until the target detection task is completed and / or the UAV reaches the target position. The target detection model is trained based on an improved YOLOv7 network model.

[0099] In some embodiments of this application, the global path planning module 11 is specifically used to extract aviable area and obstacle information from environmental map data, and to extract takeoff position, target position and flight constraints from flight mission data; using the takeoff position as the root node of the path tree and the target position as the target node, a random sampling point is generated in the aviable area, and the nearest neighbor node to the random sampling point is found in the path tree; a new node is generated by expanding from the neighbor node to the random sampling point by a first preset step size; the path status from the neighbor node to the new node is checked according to the obstacle information, and when an obstacle exists, the new node is discarded and the process returns to the step of generating a random sampling point in the aviable area; when no obstacle exists, the new node is added to the path tree; all neighbor nodes within the neighborhood of the new node are found in the path tree, and for any one of the neighbor nodes, the first path cost when the root node is connected to the new node through the neighbor node is calculated, and the neighbor node corresponding to the minimum first path cost is selected as the parent node of the new node; Calculate the second path cost when the root node connects to neighboring nodes through the new node. If the second path cost is lower than the original path cost of the corresponding neighboring node, replace the parent node of the neighboring node with the new node. Set all nodes in the path tree except the root node as leaf nodes, update the path cost of each leaf node in the path tree, and return to the step of generating a random sampling point in the flyable area. After each new node is added to the path tree, if the distance between the added new node and the target node is less than a preset threshold, backtrack from the target node to the root node according to the path tree to generate an initial path. Smooth the initial path and adjust it based on flight constraints to obtain the global flight path.

[0100] In some embodiments of this application, the global path planning module 11 is also specifically used to extract the path to be updated from the root node to the leaf node for any leaf node; and to calculate the path cost of the path to be updated based on the path length, path curvature and the shortest distance between the path to be updated and the obstacle.

[0101] In some embodiments of this application, the local obstacle avoidance module 12 is specifically used to detect obstacles in a preset area around the UAV based on a target detection model. When an obstacle is detected, the obstacle is tracked. The tracking process includes continuously collecting obstacle dynamic information and continuously acquiring the current flight status information of the UAV. The distance between the obstacle and the UAV over time is determined based on the current flight status information and the obstacle dynamic information. If the distance between the UAV and the obstacle is always greater than a preset safe obstacle avoidance threshold, the tracking of the obstacle is stopped. If the distance between the UAV and the obstacle is less than the preset safe obstacle avoidance threshold at any time, it is determined that there is a risk of conflict between the UAV and the obstacle.

[0102] In some embodiments of this application, the UAV cruise detection device further includes a deep reinforcement learning model training module, used to design a first composite reward function with obstacle avoidance as the goal; and to design a second composite reward function with returning the global path as the goal; wherein the weight allocation of the second composite reward function is different from that of the first composite reward function.

[0103] In some embodiments of this application, the UAV cruise detection device further includes a target detection model training module for constructing an improved YOLOv7 network model. The improved YOLOv7 network model uses the YOLOv7 network structure as its basic framework, embedding a CBAM module after the ELAN module in the backbone feature extraction network, replacing the SPPCSPC module in the neck feature fusion network with an ASPP module, and changing the bounding box regression loss function to the SIOU loss function. The device acquires a UAV aerial photography dataset and converts it to YOLO format, dividing it into training and validation sets. The improved YOLOv7 network model is then trained and validated using the training and validation sets to obtain the final target detection model.

[0104] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0105] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0106] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 2 of the embodiment includes: at least one processor 20 ( Figure 5 (Only one is shown in the image) a processor, a memory 21, and a computer program 22 stored in the memory 21 and capable of running on the at least one processor 20. When the processor 20 executes the computer program 22, it implements the steps in the above embodiments of the anomaly detection method for various power grid operation and maintenance images.

[0107] Electronic device 2 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Electronic device 2 may include, but is not limited to, processor 20 and memory 21. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 2 and does not constitute a limitation on electronic device 2. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0108] The processor 20 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or it can be any conventional processor.

[0109] In some embodiments, memory 21 may be an internal storage unit of electronic device 2, such as a hard disk or memory of electronic device 2. In other embodiments, memory 21 may be an external storage device of electronic device 2, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, etc., equipped on electronic device 2. Furthermore, memory 21 may include both internal and external storage units of electronic device 2. Memory 21 is used to store operating system, application programs, bootloader, data, and other programs, such as program code of computer programs. Memory 21 may also be used to temporarily store data that has been output or will be output.

[0110] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above embodiments of the UAV cruise detection method.

[0111] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to execute the steps described in the various UAV cruise detection method embodiments.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0113] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting unmanned aerial vehicle (UAV) cruise operations, characterized in that, include: The environmental map data and flight mission data corresponding to the target detection task are acquired, and the global path planning is performed using the improved fast expanding random tree algorithm to obtain the global flight path of the UAV. The system detects a preset area around the drone. When an obstacle is detected that poses a risk of conflict with the drone, a preset deep reinforcement learning model is used to control the drone's actions and obtain a local obstacle avoidance path. The deep reinforcement learning model uses the current flight state information and obstacle dynamic information as the state space, the drone's action commands as the action space, and the goal of returning to the global flight path after obstacle avoidance to design a dynamic reward function. The UAV is controlled to fly along the global flight path and the local obstacle avoidance path, and during the flight along the global flight path, the UAV performs the target detection task based on a preset target detection model until the target detection task is completed and / or the UAV reaches the target position; wherein, the target detection model is trained based on an improved YOLOv7 network model; The steps of acquiring environmental map data and flight mission data corresponding to the target detection task, and using the improved fast extended random tree algorithm for global path planning to obtain the global flight path of the UAV include: The flightable area and obstacle information are extracted from the environmental map data, and the takeoff position, target position and flight constraints are extracted from the flight mission data. Using the takeoff position as the root node of the path tree and the target position as the target node, a random sampling point is generated in the flyable area, and the nearest neighbor node to the random sampling point is found in the path tree; New nodes are generated by extending the random sampling points from the neighboring nodes by a first preset step size. Based on the obstacle information, check the path status from the neighboring nodes to the new node. If an obstacle exists, discard the new node and return to the step of generating a random sampling point in the flyable area; if no obstacle exists, add the new node to the path tree. Find all neighboring nodes within the neighborhood range of the new node in the path tree. For any one of the neighboring nodes, calculate the first path cost when the root node is connected to the new node through the neighboring node. Select the neighboring node with the minimum first path cost as the parent node of the new node. Calculate the second path cost when the root node is connected to the neighboring node through the new node. If the second path cost is lower than the original path cost of the corresponding neighboring node, replace the parent node of the neighboring node with the new node. Set all nodes in the path tree except the root node as leaf nodes, update the path cost of each leaf node in the path tree, and return to the step of generating a random sampling point in the flyable area; After each new node is added to the path tree, if the distance between the added new node and the target node is less than a preset threshold, an initial path is generated by backtracking from the target node to the root node according to the path tree. The initial path is smoothed and adjusted based on the flight constraints to obtain the global flight path; The step of updating the path cost of each leaf node in the path tree includes: For any leaf node, extract the path to be updated from the root node to the leaf node; The path cost of the path to be updated is calculated based on the path length, path curvature, and the shortest distance between the path to be updated and the obstacle.

2. The UAV cruise detection method according to claim 1, characterized in that, The step of detecting the preset area around the drone includes: Based on the target detection model, obstacles are detected in a preset area around the UAV. When an obstacle is detected, the obstacle is tracked. The tracking process includes continuously collecting the obstacle's dynamic information and continuously acquiring the UAV's current flight status information. Based on the current flight status information and the obstacle dynamic information, the relationship between the distance between the obstacle and the drone over time is determined. If the distance between the drone and the obstacle is always greater than a preset safe obstacle avoidance threshold, the tracking of the obstacle is stopped. If the distance between the drone and the obstacle is less than the preset safe obstacle avoidance threshold at any time, it is determined that there is a risk of conflict between the drone and the obstacle.

3. The UAV cruise detection method according to claim 1, characterized in that, The dynamic reward function is a weighted composite reward function, and the step of designing the dynamic reward function with the goal of returning to the global path after obstacle avoidance includes: The first composite reward function is designed with obstacle avoidance as the objective. A second composite reward function is designed with the goal of returning the global path; wherein the weight allocation of the second composite reward function is different from that of the first composite reward function.

4. The UAV cruise detection method according to claim 3, characterized in that, The expression for the dynamic reward function is: ； in, , , , , and All are weighting coefficients. Indicates the total reward. The reward indicates that the drone has successfully reached the target location. This indicates the penalty for a drone colliding with an obstacle. This represents the progressive reward for the drone's distance from the intermediate target point. This indicates the penalty for the drone staying for an extended period of time. This indicates a reward for the drone's safe obstacle avoidance. This indicates a reward or penalty for drone speed control; the intermediate target point is a node position where the drone advances a second preset step length along the global flight path from its current position.

5. The UAV cruise detection method according to claim 1, characterized in that, The training steps of the object detection model include: An improved YOLOv7 network model is constructed. The improved YOLOv7 network model adopts the YOLOv7 network structure as the basic framework, embeds the CBAM module after the ELAN module in the backbone feature extraction network, replaces the SPPCSPC module in the neck feature fusion network with the ASPP module, and changes the bounding box regression loss function to the SIOU loss function. Acquire the drone aerial photography dataset and convert it to YOLO format, then divide it into training and validation sets; The improved YOLOv7 network model is trained and validated using the training and validation sets to obtain the final object detection model.

6. A drone cruise detection device, characterized in that, include: The global path planning module is used to acquire environmental map data and flight mission data corresponding to the target detection task, and to perform global path planning using an improved fast expanding random tree algorithm to obtain the global flight path of the UAV. The local obstacle avoidance module is used to detect a preset area around the UAV. When an obstacle is detected that poses a risk of conflict with the UAV, a preset deep reinforcement learning model is used to control the UAV's actions and obtain a local obstacle avoidance path. The deep reinforcement learning model uses the current flight state information and obstacle dynamic information as the state space, the UAV action commands as the action space, and the goal of returning to the global flight path after obstacle avoidance to design a dynamic reward function. The cruise detection module is used to control the UAV to fly along the global aircraft path and the local obstacle avoidance path, and to perform the target detection task based on a preset target detection model during the flight along the global aircraft path, until the target detection task is completed and / or the UAV reaches the target position; wherein, the target detection model is trained based on an improved YOLOv7 network model; The UAV cruise detection device employs the UAV cruise detection method as described in any one of claims 1 to 5.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the UAV cruise detection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the UAV cruise detection method as described in any one of claims 1 to 5.

Images