Low-altitude UAV intelligent inspection system and method based on deep learning

Abstract

The present invention discloses a low-altitude unmanned aerial vehicle (UAV) intelligent inspection system and method based on deep learning, which relates to the field of UAV intelligent control technology. By pre-collecting a UAV information set and a task information set, an anomaly detection model is constructed based on the task information set, and real-time flight parameter data of the UAV is collected. Based on the anomaly detection model, the real-time flight parameter data and the UAV information set, a parameter adjustment optimization model is constructed, the flight parameter adjustment strategy is solved and output, the flight parameter adjustment strategy is sent to the UAV, and anomaly detection information collected during the UAV flight is collected in real time. Dynamic parameter adjustment is performed based on the anomaly detection information and the UAV information set; the intelligence level and operational efficiency of the UAV inspection are significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of drone intelligent control technology, and in particular to a low-altitude drone intelligent inspection system and method based on deep learning. Background Art

[0002] Low-altitude drone inspections have become a highly effective tool for modern infrastructure monitoring and agricultural management. During inspections, drones must flexibly respond to dynamic environmental changes and a variety of abnormalities. These anomalies are often uncertain and can vary in location, form, and severity, such as damaged power lines, minor cracks in bridges, and pests and diseases in farmland. These anomalies are often fluid, and traditional static flight parameter settings are unable to cope with such complex and changing inspection requirements.

[0003] Existing inspection technologies often rely on pre-set flight paths and parameters. These static configurations exhibit significant deficiencies in the face of real-time changes. Static parameters can cause drones to miss critical details when detecting anomalies because they are unable to adjust their flight altitude, speed, and path based on emerging anomalies. Static path planning can also lead to wasted energy and reduced inspection efficiency because it cannot avoid obstacles or prioritize important areas in real time. Furthermore, static parameter settings ignore real-time changes in environmental factors such as weather, wind speed, and lighting, which can significantly affect the effectiveness and safety of drone inspections.

[0004] A Chinese patent with authorization publication number CN116071962B discloses an automatic highway drone inspection system, including a drone airport for realizing drone takeoff and landing operations, an information collection module for collecting image information and location information of the road section to be inspected, a highway maintenance operation platform for estimating the repair time of the diseased road section based on the image information of the road section to be inspected, and for matching the location information of the diseased road section with the corresponding estimated repair time, and a task management module for replanning the drone inspection route based on the location information of the diseased road section and the corresponding estimated repair time information. The inspection route is replanned through the combined effects of data measurement, parameter calculation and repair time estimation of the diseased road section; however, this method fails to solve the problem of dynamic parameter control during flight.

[0005] To this end, the present invention proposes a low-altitude UAV intelligent inspection system and method based on deep learning. Summary of the Invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a low-altitude UAV intelligent inspection system and method based on deep learning, which significantly improves the intelligence level and operational efficiency of UAV inspections.

[0007] To achieve the above objectives, a low-altitude UAV intelligent inspection method based on deep learning is proposed, which includes the following steps:

[0008] Step 1: Collect drone information and mission information in advance;

[0009] Step 2: Build an anomaly detection model based on the task information set;

[0010] Step 3: Collect real-time flight parameter data of the UAV, build a parameter adjustment optimization model based on the anomaly detection model, real-time flight parameter data and UAV information collection, and solve and output the flight parameter adjustment strategy;

[0011] Step 4: Send flight parameter adjustment strategies to the drone and collect anomaly detection information collected during the drone flight in real time;

[0012] Step 5: Dynamic parameter adjustment based on anomaly detection information and drone information collection.

[0013] The drone information set is collected in the following manner:

[0014] According to the actual hardware configuration of the UAV, the hardware parameters of the UAV are counted as the UAV information set;

[0015] The task information set is collected in the following manner:

[0016] Collect all preset inspection scenarios and corresponding inspection task types;

[0017] For each type of inspection task in each inspection scenario, sample images that have been labeled and position-annotated are collected in advance to form a task information set.

[0018] The method of constructing anomaly detection model based on task information set is as follows:

[0019] For each inspection scenario, the corresponding inspection task type is:

[0020] Read the sample image set corresponding to the inspection task type from the task information set;

[0021] Divide the sample image set into training set, validation set and test set;

[0022] Select the YOLO model or YOLOv3 model as the anomaly detection model;

[0023] Resize all sample images to a uniform size to fit the input requirements of the anomaly detection model;

[0024] Apply data augmentation techniques such as random rotation, flipping, and scaling to sample images to improve the generalization ability of the model;

[0025] Pre-set the hyperparameter set of the anomaly detection model;

[0026] The anomaly detection model uses a weighted sum of the localization loss of bounding box regression, the classification loss of cross entropy loss, and the confidence loss of target box matching as the loss function;

[0027] Use the training set to train the anomaly detection model and perform forward propagation to calculate the prediction results;

[0028] Calculate the loss and optimize the model parameters of the anomaly detection model through backpropagation;

[0029] Evaluate the model on the validation set, check for overfitting, and manually adjust hyperparameters until the anomaly detection model reaches the preset accuracy on the validation set;

[0030] The output of the anomaly detection model includes bounding box information, confidence score and category label for each detection box in the sample image;

[0031] The anomaly detection model is further validated on the test set to further verify the model performance, and the accuracy, recall and F1 score are recorded.

[0032] The method of collecting real-time flight parameter data of the UAV is:

[0033] According to the needs, set the actual inspection scene and inspection task type for the drone in the background of the drone;

[0034] When controlling the drone to actually perform inspection tasks, the drone's built-in image capture equipment, sensors and navigation system are used to collect various flight parameters in real time to form real-time flight parameter data.

[0035] The method of constructing a parameter adjustment optimization model based on the anomaly detection model, real-time flight parameter data, and drone information collection to solve and output a flight parameter adjustment strategy includes the following steps:

[0036] Step 31: Input the real-time image captured by the drone's image capture device into the anomaly detection model. If the confidence level output by the anomaly detection model is greater than a preset confidence threshold, it is determined that an anomaly exists and the process proceeds to step 32. If the confidence level output is less than or equal to the preset confidence threshold, no processing is performed.

[0037] Step 32: Determine the actual geographic coordinates of the abnormal location where the abnormality occurred in the inspection scene based on the position of the bounding box output by the anomaly detection model in the real-time image and the real-time position of the drone;

[0038] The method for determining the actual geographic coordinates of the abnormal location where the abnormality occurs in the inspection scene is:

[0039] Get the anomaly bounding box data (x) from the output of the YOLO model bbox ,y bbox , w bbox , h bbox ); where x bbox Indicates the center horizontal coordinate of the bounding box, y bbox Indicates the center ordinate of the bounding box, w bbox Indicates the width of the bounding box, h bbox Indicates the height of the bounding box;

[0040] Mark the spatial coordinates of the drone's current position as (xc, yc, zc);

[0041] Use the drone's flight altitude zc and the camera's viewing angle parameters to convert the image plane coordinates into ground coordinates;

[0042] Denote the viewing angle of the drone’s image capture device as θ and calculate the conversion coefficient SC from pixels to actual distance;

[0043] Calculate the geographic abscissa xr of the abnormal position according to the central abscissa and the conversion coefficient;

[0044] Calculate the geographic ordinate yr of the anomaly location based on the central ordinate and the conversion coefficient;

[0045] Then the actual geographical coordinates of the abnormal position are expressed as (geographic abscissa, geographic ordinate);

[0046] Step 33: Calculate the anomaly impact factor e based on the confidence level output by the anomaly detection model and the actual coverage area of ​​the anomaly location;

[0047] Step 34: Calculate the abnormal distance factor d based on the real-time position of the UAV and the abnormal position;

[0048] Step 35: Calculate the path adjustment coefficient g based on the current real-time flight speed and flight altitude of the UAV;

[0049] Step 36: Adjust the flight speed according to the abnormal impact factor e;

[0050] Step 37: Adjust the flight altitude according to the abnormal distance factor d;

[0051] Step 38: Adjust the path offset ΔP according to the path adjustment proportional coefficient g; the calculation formula of the path offset ΔP is: ΔP = g × (xc-xr, yc-yr);

[0052] Step 39: The abnormal impact factor e, the abnormal distance factor d, and the path adjustment proportional coefficient g are used to form a flight parameter adjustment strategy.

[0053] The method of collecting abnormal detection information collected during the flight of the drone in real time is:

[0054] During the inspection process after the UAV adjusts the strategy based on the flight parameters, it continues to collect the abnormal location, abnormal category, confidence level, and abnormal area as abnormality detection information;

[0055] The method of dynamic parameter adjustment based on anomaly detection information and drone information collection is as follows:

[0056] If the difference between the real-time confidence and the confidence at the previous unit time is greater than the preset confidence difference threshold, or the difference between the area covered by the real-time abnormal position and the area covered by the abnormal position at the previous unit time is greater than the preset area difference threshold, the flight speed and flight altitude are dynamically adjusted.

[0057] A low-altitude UAV intelligent inspection system based on deep learning is proposed, which includes an information collection module, a detection model construction module, a strategy static adjustment module, and a strategy dynamic adjustment module. The modules are electrically connected to each other.

[0058] The information collection module collects the drone information set and the mission information set in advance, sends the mission information set to the detection model construction module, and sends the drone information set to the strategy static adjustment module;

[0059] The detection model building module builds an anomaly detection model based on the task information set and sends the anomaly detection model to the strategy static adjustment module;

[0060] The strategy static adjustment module collects real-time flight parameter data of the UAV, builds a parameter adjustment optimization model based on the anomaly detection model, real-time flight parameter data and UAV information collection, solves and outputs the flight parameter adjustment strategy, sends the flight parameter adjustment strategy to the UAV, and collects anomaly detection information collected during the UAV flight in real time, and sends the collected anomaly detection information to the strategy dynamic adjustment module;

[0061] The strategy dynamic adjustment module performs dynamic parameter adjustment based on anomaly detection information and drone information collection.

[0062] An electronic device is proposed, comprising: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;

[0063] The processor executes the above-mentioned low-altitude UAV intelligent inspection method based on deep learning by calling the computer program stored in the memory.

[0064] A computer-readable storage medium is provided, on which a rewritable computer program is stored.

[0065] When the computer program runs on a computer device, the computer device executes the above-mentioned low-altitude UAV intelligent inspection method based on deep learning.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] The present invention pre-collects a set of drone information and mission information, constructs an anomaly detection model based on the mission information set, collects real-time drone flight parameter data, constructs a parameter adjustment optimization model based on the anomaly detection model, real-time flight parameter data, and drone information set, solves and outputs a flight parameter adjustment strategy, sends the flight parameter adjustment strategy to the drone, and collects anomaly detection information collected during drone flight in real time. Based on the anomaly detection information and drone information set, dynamic parameter adjustment is performed. A deep learning model is used to detect and identify anomalies in real time, and the drone's flight parameters are dynamically adjusted based on the drone's hardware and software information, such as camera resolution, navigation accuracy, and endurance. This method not only ensures that the drone maintains an appropriate observation angle and flight path during inspections, but also improves detection efficiency and accuracy. By integrating real-time data processing and parameter adjustment algorithms, the drone can automatically optimize its inspection tasks based on environmental changes, meeting diverse inspection needs and significantly improving the intelligence level and operational efficiency of drone inspections. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a flow chart of the low-altitude UAV intelligent inspection method based on deep learning in Example 1 of the present invention;

[0069] Figure 2 This is a module connection diagram of the low-altitude UAV intelligent inspection system based on deep learning in Example 2 of the present invention. DETAILED DESCRIPTION

[0070] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0071] Example 1

[0072] like Figure 1 As shown in the figure, the low-altitude UAV intelligent inspection method based on deep learning includes the following steps:

[0073] Step 1: Collect drone information and mission information in advance;

[0074] Step 2: Build an anomaly detection model based on the task information set;

[0075] Step 3: Collect real-time flight parameter data of the UAV, build a parameter adjustment optimization model based on the anomaly detection model, real-time flight parameter data and UAV information collection, and solve and output the flight parameter adjustment strategy;

[0076] Step 4: Send flight parameter adjustment strategies to the drone and collect anomaly detection information collected during the drone flight in real time;

[0077] Step 5: Dynamic parameter adjustment based on anomaly detection information and drone information collection.

[0078] The drone information set is collected in the following manner:

[0079] According to the actual hardware configuration of the UAV, the hardware parameters of the UAV are counted as the UAV information set;

[0080] Specifically, the drone information set includes hardware information such as camera resolution, sensor type and accuracy, maximum flight time, maximum flight speed, and maximum load capacity, as well as software information such as navigation system accuracy and real-time data processing capabilities.

[0081] Furthermore, the task information set is collected in the following manner:

[0082] Collect all preset inspection scenarios and corresponding inspection task types; it is understandable that the specific inspection tasks that need to be performed are different for different inspection scenarios, for example: the inspection scenarios may include power lines, bridges, farmlands, etc., and the inspection task types corresponding to each inspection scenario may be damage, cracks, pests and diseases, etc., that is, when the inspection scenario is a power line, drone inspection can be used to detect whether the line is damaged. Similarly, when the bridge is used as the inspection scenario, drone inspection can be used to detect whether the bridge has cracks. When the farmland is used as the inspection scenario, drone inspection can be used to detect whether the crops have pests and diseases. Therefore, in different inspection tasks, the abnormal situations that the drone needs to identify are different; it should be noted that the above-mentioned inspection scenarios listed here in this embodiment are only some typical examples. In actual application, they can be further expanded. The expansion of inspection scenarios and the expansion of inspection tasks are within the protection scope of the present invention.

[0083] For each type of inspection task in each inspection scenario, several sample images with labeled and position annotations are collected in advance to form a task information set;

[0084] It should be noted that the collected sample images should ensure that the sample data covers different environments (such as weather conditions, seasonal changes) and perspectives, and preferably be image data collected from historical inspection tasks, public datasets or manual shooting.

[0085] Specifically, the label annotation refers to marking the existing anomalies in the sample image, and the anomaly is the anomaly type corresponding to the inspection task, such as damage, cracks, pests and diseases, etc.; for sample images with anomalies, the label annotation is set to 1, and for sample images without anomalies, the label annotation is set to 0.

[0086] The position annotation refers to providing abnormal bounding box information for a sample image with an abnormality, and the bounding box information includes the coordinates of the center point of the bounding box, the width and height of the bounding box.

[0087] Furthermore, the method of constructing an anomaly detection model based on the task information set is:

[0088] For each inspection scenario, the corresponding inspection task type is:

[0089] Read the sample image set corresponding to the inspection task type from the task information set;

[0090] Divide the sample image set into training set, validation set and test set; generally, the division ratio of training set, validation set and test set is 70%, 15% and 15%;

[0091] Select the YOLO model or YOLOv3 model as the anomaly detection model;

[0092] Resize all sample images to a uniform size to fit the input requirements of the anomaly detection model;

[0093] Apply data augmentation techniques such as random rotation, flipping, and scaling to sample images to improve the generalization ability of the model;

[0094] Presetting a set of hyperparameters for the anomaly detection model; the hyperparameters include learning rate, batch size, number of training rounds, etc.;

[0095] The anomaly detection model uses a weighted sum of the localization loss of bounding box regression, the classification loss of cross entropy loss, and the confidence loss of target box matching as the loss function;

[0096] Use the training set to train the anomaly detection model and perform forward propagation to calculate the prediction results;

[0097] Calculate the loss and optimize the model parameters of the anomaly detection model through backpropagation;

[0098] Evaluate the model on the validation set, check for overfitting, and manually adjust hyperparameters until the anomaly detection model reaches the preset accuracy on the validation set;

[0099] The output of the anomaly detection model includes bounding box information, confidence score and category label for each detection box in the sample image;

[0100] The anomaly detection model is further validated on the test set to further verify the model performance, and the accuracy, recall and F1 score are recorded.

[0101] Furthermore, the method of collecting real-time flight parameter data of the drone is:

[0102] According to the needs, set the actual inspection scene and inspection task type for the drone in the background of the drone;

[0103] When controlling the drone to actually perform inspection tasks, the drone's built-in image capture devices, sensors and navigation systems, such as the onboard camera, IMU (inertial measurement unit) and GPS module, are used to collect various flight parameters in real time to form real-time flight parameter data.

[0104] The flight parameters include: real-time images captured by the image capture device, current flight speed captured by the sensor and navigation system, current flight altitude, GPS coordinates including longitude, latitude and altitude, attitude information including pitch angle, yaw angle and roll angle, battery status including battery percentage and discharge rate, etc.

[0105] Furthermore, the construction of a parameter adjustment optimization model based on the anomaly detection model, real-time flight parameter data, and drone information set, and the solution of the output flight parameter adjustment strategy include the following steps:

[0106] Step 31: Input the real-time image captured by the drone's image capture device into the anomaly detection model. If the confidence level output by the anomaly detection model is greater than a preset confidence threshold, it is determined that an anomaly exists and the process proceeds to step 32. If the confidence level output is less than or equal to the preset confidence threshold, no processing is performed.

[0107] Step 32: Determine the actual geographic coordinates of the abnormal location where the abnormality occurred in the inspection scene based on the position of the bounding box output by the anomaly detection model in the real-time image and the real-time position of the drone;

[0108] Preferably, the method for determining the actual geographical coordinates of the abnormal location where the abnormality occurs in the inspection scene is:

[0109] Get the anomaly bounding box data (x) from the output of the YOLO model bbox ,y bbox , w bbox , hbbox ); where x bbox Indicates the center horizontal coordinate of the bounding box, y bbox Indicates the center ordinate of the bounding box, w bbox Indicates the width of the bounding box, h bbox Indicates the height of the bounding box;

[0110] Mark the spatial coordinates of the drone's current position as (xc, yc, zc);

[0111] Use the drone's flight altitude zc and the camera's viewing angle parameters such as focal length and field of view to convert the image plane coordinates into ground coordinates;

[0112] The viewing angle of the drone's image capture device is denoted as θ, and the conversion coefficient SC from pixels to actual distance is calculated. The conversion coefficient SC is calculated as follows: Among them, Iw represents the width of the image captured by the image capture device, in pixels; it is used to convert the image coordinates of the bounding box into actual geographic coordinates in the field of view of the drone; since the drone camera can be simplified to a pinhole camera model, this is a commonly used model in computer vision. This model assumes that all light passes through a single point and forms an image on the sensor, and this image is a two-dimensional projection of the three-dimensional world. The size of the object in the image depends on the distance from the camera. The closer the object is to the camera, the larger it appears in the image. This change is described by the perspective projection relationship, and the height directly affects the actual distance represented by a pixel on the ground. For a fixed field of view angle, the higher the flight altitude, the larger the actual area covered by a pixel on the ground; and the viewing angle θ is the maximum viewing angle range that the camera lens can see. The field of view angle determines the width of the ground area that the camera can see; therefore, based on the flight altitude and viewing angle, and based on the similarity of triangles, the conversion coefficients between coordinate systems can be derived;

[0113] According to the central horizontal coordinate and the conversion coefficient, the geographic horizontal coordinate xr of the abnormal position is calculated; specifically, the calculation formula of the geographic horizontal coordinate xr is:

[0114] According to the central ordinate and the conversion coefficient, the geographic ordinate yr of the abnormal position is calculated; specifically, the calculation formula of the geographic ordinate yr is: Ih represents the height of the image captured by the image capture device, in pixels;

[0115] The actual geographic coordinates of the abnormal position are expressed as (geographic horizontal coordinate, geographic vertical coordinate). Since the target location of a general inspection task is the ground or a lower scene, the actual height of the abnormal position can be set to 0.

[0116] Step 33: Calculate the anomaly impact factor e based on the confidence level output by the anomaly detection model and the actual coverage area of ​​the anomaly location. The actual coverage area is the mapping of the bounding box in the actual geographic coordinates, that is, the area can be calculated as w bbox ×SC×h bbox ×SC; the calculation formula of the abnormal impact factor e can be: e = c × s × t, where c is the confidence level, s is the actual coverage area, and t is the impact weight of the preset inspection task type. The more important the inspection task, the larger the t value;

[0117] Step 34: Calculate the abnormal distance factor d based on the real-time position of the UAV and the abnormal position; specifically, the calculation formula of the abnormal distance factor d is: d=zc;

[0118] Step 35: Calculate the path adjustment coefficient g based on the current real-time flight speed and flight altitude of the UAV. The calculation formula of the path adjustment coefficient g is: Where k is the preset sensitivity constant, v is the real-time flight speed of the UAV, and h is the real-time altitude of the UAV;

[0119] Step 36: Adjust the flight speed based on the abnormality impact factor e. The flight speed is adjusted by adjusting the flight speed v to v-k1×e, where k1 is a preset proportional coefficient. It is understood that when the confidence level, the range of abnormality coverage, or the impact weight is larger, the abnormality is more serious and the flight speed needs to be reduced to ensure that the drone can fully observe and confirm the abnormality.

[0120] Step 37: Adjust the flight altitude based on the anomaly distance factor d. The flight altitude is adjusted by adjusting the flight altitude zc to zc-k2×d, where k2 is a preset proportional coefficient. It is understood that the farther the distance from the anomaly, the greater the altitude reduction, so as to get closer to the anomaly location, thereby ensuring higher clarity and resolution of observation.

[0121] Step 38: Adjust the path offset ΔP based on the path adjustment proportional coefficient g. The calculation formula for the path offset ΔP is: ΔP = g × (xc - xr, yc - yr). It can be understood that the path offset ΔP refers to shifting the UAV's flight path toward the location of the abnormality. The greater the flight speed or the lower the flight altitude, the smaller the offset, to prevent excessive adjustment from passing over the abnormal location and causing omission of the abnormality.

[0122] Step 39: The abnormal impact factor e, the abnormal distance factor d, and the path adjustment proportional coefficient g are used to form a flight parameter adjustment strategy.

[0123] Furthermore, sending the flight parameter adjustment strategy to the drone is to send the flight parameter adjustment strategy to the drone control system, and the drone control system regulates the flight speed, flight altitude and flight path of the drone according to the flight parameter adjustment strategy.

[0124] The method of collecting abnormal detection information collected during the flight of the drone in real time is as follows:

[0125] During the inspection process after the UAV adjusts the strategy based on the flight parameters, it continues to collect the abnormal location, abnormal category, confidence level, and abnormal area as abnormality detection information;

[0126] It is understandable that even after the drone adjusts its flight parameters according to its strategy, there may still be sudden changes due to changes in wind speed and direction, aggravation of abnormal conditions, obstacle avoidance needs, etc., so dynamic parameter adjustments need to be made in real time according to the actual environment;

[0127] Specifically, the method of dynamic parameter adjustment based on anomaly detection information and drone information collection is as follows:

[0128] If the difference between the real-time confidence and the confidence at the previous unit time is greater than the preset confidence difference threshold, or the difference between the area covered by the real-time abnormal position and the area covered by the abnormal position at the previous unit time is greater than the preset area difference threshold, the flight speed and flight altitude are dynamically adjusted.

[0129] Specifically, the method of dynamically adjusting the flight speed and flight altitude is:

[0130] Adjust the flight speed to min(vmax, v×(1+Δc×Δs×t)); where min(·) is the minimum value function, vmax is the maximum flight speed of the drone, Δc is the confidence difference, and Δs is the difference between the area. That is, the larger the difference, the faster the flight to the abnormal location.

[0131] Adjust the flight altitude to Among them, cr is the resolution of the real-time image, and α is the preset proportional coefficient. That is, when the difference is larger, it means that the position of the drone needs to be lowered as much as possible in order to capture the abnormal position more clearly.

[0132] Example 2

[0133] like Figure 2 As shown, the low-altitude UAV intelligent inspection system based on deep learning includes an information collection module, a detection model construction module, a strategy static adjustment module, and a strategy dynamic adjustment module; wherein each module is electrically connected;

[0134] The information collection module collects the drone information set and the mission information set in advance, sends the mission information set to the detection model construction module, and sends the drone information set to the strategy static adjustment module;

[0135] The detection model building module builds an anomaly detection model based on the task information set and sends the anomaly detection model to the strategy static adjustment module;

[0136] The strategy static adjustment module collects real-time flight parameter data of the UAV, builds a parameter adjustment optimization model based on the anomaly detection model, real-time flight parameter data and UAV information collection, and solves and outputs the flight parameter adjustment strategy; sends the flight parameter adjustment strategy to the UAV, and collects anomaly detection information collected during the UAV flight in real time, and sends the collected anomaly detection information to the strategy dynamic adjustment module;

[0137] The strategy dynamic adjustment module performs dynamic parameter adjustment based on anomaly detection information and drone information collection.

[0138] Example 3

[0139] According to another aspect of the present application, an electronic device is provided. The electronic device may include one or more processors and one or more memories. The memories may store computer-readable code that, when executed by the one or more processors, may execute the deep learning-based low-altitude drone intelligent inspection method described above.

[0140] The electronic device may include a bus, one or more CPUs, ROM, RAM, a communication port connected to a network, input / output components, a hard disk, etc. A storage device in the electronic device, such as a ROM or a hard disk, may store the low-altitude UAV intelligent inspection method based on deep learning provided in this application.

[0141] Furthermore, the electronic device may further include a user interface. Of course, this architecture is merely exemplary, and when implementing different devices, one or more components in the electronic device may be omitted according to actual needs.

[0142] Example 4

[0143] A computer-readable storage medium according to an embodiment of the present application. Computer-readable instructions are stored on the computer-readable storage medium. When the computer-readable instructions are executed by a processor, the low-altitude drone intelligent inspection method based on deep learning according to the embodiment of the present application described with reference to the above figures can be executed. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0144] In addition, according to embodiments of the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the present application provides a non-transitory machine-readable storage medium storing machine-readable instructions capable of being executed by a processor to execute instructions corresponding to the steps of the method provided in the present application. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the method of the present application are performed.

[0145] The methods, apparatuses, and devices of the present application may be implemented in many ways. For example, the methods, apparatuses, and devices of the present application may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present application are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present application may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present application. Therefore, the present application also covers recording media that store programs for executing the methods according to the present application.

[0146] In addition, the parts of the above technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0147] The above-described specific embodiments further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above description is merely a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

[0148] The above preset parameters or preset thresholds are all set by those skilled in the art according to actual conditions or obtained through large amounts of data simulation.

[0149] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A low-altitude UAV intelligent inspection method based on deep learning, characterized by: The following steps are involved: Step 1: Collect drone information and mission information in advance; Step 2: Build an anomaly detection model based on the task information set; select the YOLO model or YOLOv3 model as the anomaly detection model; Step 3: Collect real-time flight parameter data of the UAV, build a parameter adjustment optimization model based on the anomaly detection model, real-time flight parameter data and UAV information collection, and solve and output the flight parameter adjustment strategy; The flight parameter adjustment strategy includes abnormal impact factor, abnormal distance factor and path adjustment proportional coefficient; According to the confidence level output by the anomaly detection model and the actual coverage area of ​​the anomaly location, the anomaly impact factor e is calculated. The calculation formula of the anomaly impact factor e is: , where c is the confidence level, s is the actual coverage area, and t is the influence weight of the preset inspection task type; According to the real-time position and abnormal position of the UAV, the abnormal distance factor d is calculated; the calculation formula of the abnormal distance factor d is: zc, zc is the flight altitude of the UAV; Calculate the path adjustment coefficient based on the drone's current real-time flight speed and flight altitude ; The path adjustment proportional coefficient The calculation formula is: ; Where k is the preset sensitivity constant, h is the real-time altitude of the UAV, and v is the real-time flight speed of the UAV; Step 4: Send flight parameter adjustment strategies to the drone and collect anomaly detection information collected during the drone flight in real time; Step 5: Dynamic parameter adjustment based on anomaly detection information and drone information collection; The method of dynamic parameter adjustment based on anomaly detection information and drone information collection is as follows: If the difference between the confidence level of the anomaly detection model detected in real time and the confidence level of the previous unit time is greater than a preset confidence difference threshold, or the difference between the area covered by the real-time anomaly location and the area covered by the anomaly location of the previous unit time is greater than a preset area difference threshold, the flight speed and altitude are dynamically adjusted; the anomaly location is the location where the anomaly occurs in the inspection scene; Adjust the flight speed to ;in, is the minimum value function, vmax is the maximum flight speed of the UAV, is the confidence difference, is the difference between the area values; v is the real-time flight speed of the UAV, and t is the influence weight of the preset inspection task type; Adjust the flight altitude to ; Where cr is the resolution of the real-time image, is the preset proportional coefficient, and zc is the flight altitude of the UAV.

2. The low-altitude UAV intelligent inspection method based on deep learning according to claim 1 is characterized in that: The drone information set is collected in the following manner: According to the actual hardware configuration of the UAV, the hardware parameters of the UAV are counted as the UAV information set; The task information set is collected in the following manner: Collect all preset inspection scenarios and corresponding inspection task types; For each type of inspection task in each inspection scenario, sample images that have been labeled and position-annotated are collected in advance to form a task information set.

3. The low-altitude UAV intelligent inspection method based on deep learning according to claim 2 is characterized in that: The method of constructing anomaly detection model based on task information set is as follows: For each inspection scenario, the corresponding inspection task type is: Read the sample image set corresponding to the inspection task type from the task information set; Divide the sample image set into training set, validation set and test set; Resize all sample images to a uniform size to fit the input requirements of the anomaly detection model; Apply data augmentation techniques such as random rotation, flipping, and scaling to sample images to improve the generalization ability of the model; Pre-set the hyperparameter set of the anomaly detection model; The anomaly detection model uses a weighted sum of the localization loss of bounding box regression, the classification loss of cross entropy loss, and the confidence loss of target box matching as the loss function; Use the training set to train the anomaly detection model and perform forward propagation to calculate the prediction results; Calculate the loss and optimize the model parameters of the anomaly detection model through backpropagation; Evaluate the model on the validation set, check for overfitting, and manually adjust hyperparameters until the anomaly detection model reaches the preset accuracy on the validation set; The output of the anomaly detection model includes bounding box information, confidence score and category label for each detection box in the sample image; The anomaly detection model is further validated on the test set to further verify the model performance, and the accuracy, recall and F1 score are recorded.

4. The low-altitude UAV intelligent inspection method based on deep learning according to claim 3 is characterized in that: The method of collecting real-time flight parameter data of the UAV is: According to the needs, set the actual inspection scene and inspection task type for the drone in the background of the drone; When controlling the drone to actually perform inspection tasks, the drone's built-in image capture equipment, sensors and navigation system are used to collect various flight parameters in real time to form real-time flight parameter data.

5. The low-altitude UAV intelligent inspection method based on deep learning according to claim 4 is characterized in that: The method of constructing a parameter adjustment optimization model based on the anomaly detection model, real-time flight parameter data, and drone information collection to solve and output a flight parameter adjustment strategy includes the following steps: Step 31: Input the real-time image captured by the drone's image capture device into the anomaly detection model. If the confidence level output by the anomaly detection model is greater than a preset confidence threshold, it is determined that an anomaly exists and the process proceeds to step 32. If the confidence level output is less than or equal to the preset confidence threshold, no processing is performed. Step 32: Determine the actual geographic coordinates of the abnormal location where the abnormality occurred in the inspection scene based on the position of the bounding box output by the anomaly detection model in the real-time image and the real-time position of the drone; Step 33: The actual coverage area is the mapping of the bounding box in the actual geographic coordinates, that is, the area is calculated as , SC represents the conversion coefficient from calculated pixel to actual distance, w bbox Indicates the width of the bounding box, h bbox represents the height of the bounding box, Step 36: Adjust the flight speed according to the abnormal impact factor e; Step 37: Adjust the flight altitude according to the abnormal distance factor d; Step 38: Mark the spatial coordinates of the drone's current position as (xc, yc, zc); adjust the scale factor according to the path , adjust the path offset ; The path offset The calculation formula is: ; Among them, xr and yr are the actual geographic horizontal coordinate and actual geographic vertical coordinate of the abnormal position, xc represents the spatial coordinate horizontal coordinate of the current position of the drone, and yc represents the spatial coordinate vertical coordinate of the current position of the drone; Step 39: Abnormal impact factor e, abnormal distance factor d and path adjustment coefficient Constitute the flight parameter adjustment strategy.

6. The low-altitude UAV intelligent inspection method based on deep learning according to claim 5 is characterized in that: The method for determining the actual geographic coordinates of the abnormal location where the abnormality occurs in the inspection scene is: Get the anomaly bounding box data (x) from the output of the YOLO model bbox ,y bbox , w bbox , h bbox ); where x bbox Indicates the center horizontal coordinate of the bounding box, y bbox Indicates the center ordinate of the bounding box, w bbox Indicates the width of the bounding box, h bbox Indicates the height of the bounding box; Use the drone's flight altitude zc and the camera's viewing angle parameters to convert the image plane coordinates into ground coordinates; Label the viewpoint of the drone's image capture device as , calculate the conversion coefficient SC from pixel to actual distance; Calculate the geographic abscissa xr of the abnormal position according to the central abscissa and the conversion coefficient; Calculate the geographic ordinate yr of the anomaly location based on the central ordinate and the conversion coefficient; The actual geographic coordinates of the abnormal position are expressed as (geographic horizontal coordinate xr, geographic vertical coordinate yr).

7. A low-altitude UAV intelligent inspection system based on deep learning, which is used to implement the low-altitude UAV intelligent inspection method based on deep learning according to any one of claims 1 to 6, characterized in that: It includes an information collection module, a detection model building module, a strategy static adjustment module, and a strategy dynamic adjustment module; wherein each module is electrically connected; The information collection module collects the drone information set and the mission information set in advance, sends the mission information set to the detection model construction module, and sends the drone information set to the strategy static adjustment module; The detection model building module builds an anomaly detection model based on the task information set and sends the anomaly detection model to the strategy static adjustment module; The strategy static adjustment module collects real-time flight parameter data of the UAV, builds a parameter adjustment optimization model based on the anomaly detection model, real-time flight parameter data and UAV information collection, solves and outputs the flight parameter adjustment strategy, sends the flight parameter adjustment strategy to the UAV, and collects anomaly detection information collected during the UAV flight in real time, and sends the collected anomaly detection information to the strategy dynamic adjustment module; The strategy dynamic adjustment module performs dynamic parameter adjustment based on anomaly detection information and drone information collection.

8. An electronic device, characterized in that: include: processor and memory, wherein The memory stores a computer program that can be called by the processor; The processor executes the low-altitude UAV intelligent inspection method based on deep learning as described in any one of claims 1 to 6 in the background by calling the computer program stored in the memory.

9. A computer-readable storage medium, characterized in that A rewritable computer program is stored thereon; When the computer program runs on a computer device, the computer device executes the low-altitude UAV intelligent inspection method based on deep learning as described in any one of claims 1 to 6 in the background.

Images