Unmanned aerial vehicle automatic inspection route generation method and system based on visual identification

Abstract

The invention provides an unmanned aerial vehicle automatic inspection route generation method and system based on visual identification, and relates to the technical field of unmanned aerial vehicle inspection, and the method comprises the steps: collecting an inspection video stream of an inspection region through a camera device carried by an inspection unmanned aerial vehicle; extracting an inspection image frame and carrying out target detection on the inspection image frame based on a deep learning model to generate a multi-scene inspection target point; based on a camera matrix and a digital elevation model, pixel coordinates of the multi-scene inspection target point are converted into actual geographic coordinates; automatically planning a structured inspection route of the inspection unmanned aerial vehicle according to the actual geographic coordinates in combination with a path curvature constraint condition; a structured inspection route is converted into a recognizable route instruction file and is imported into a flight control system of the inspection unmanned aerial vehicle, so that high efficiency, precision, automation and intelligence of the whole inspection process of the unmanned aerial vehicle are realized by fusing visual recognition and intelligent route planning technologies, and the system adaptability, the inspection quality and the operation safety are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) inspection technology, and in particular to a method and system for generating automatic inspection routes for UAVs based on visual recognition. Background Technology

[0002] With the rapid development of drone technology, drones are increasingly being used in the inspection of infrastructure such as power, oil, and natural gas.

[0003] In practical applications, existing technologies have significant technical shortcomings. First, traditional drone inspection methods mainly rely on manually pre-setting flight paths. Operators need to manually design flight paths and shooting points based on the location and characteristics of the inspection targets. This method is not only time-consuming and labor-intensive, but also easily affected by human factors, leading to unstable inspection quality. Second, traditional drone inspection methods are difficult to adapt to complex and ever-changing inspection environments, especially when facing a large number of dispersed inspection targets, and cannot efficiently plan optimal flight paths.

[0004] Therefore, it is necessary to provide a method and system for generating automatic inspection routes for unmanned aerial vehicles based on visual recognition to solve the above-mentioned technical problems. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method and system for generating automatic inspection routes for unmanned aerial vehicles (UAVs) based on visual recognition. This system solves the problems of existing technologies being unable to intelligently generate automatic inspection routes for UAVs based on visual recognition technology, and the low efficiency and poor quality of UAV inspection data.

[0006] This invention provides a method for generating automatic inspection routes for unmanned aerial vehicles (UAVs) based on visual recognition, the method comprising: The inspection drone collects inspection video streams of the inspection area. Extract the inspection image frames from the inspection video stream, and perform target detection on the inspection image frames based on a deep learning model to generate multi-scene inspection target points; Based on the camera matrix and digital elevation model of the inspection drone, the pixel coordinates of the multi-scene inspection target points are converted into actual geographic coordinates. Based on the actual geographical coordinates of the multi-scenario inspection target points and combined with path curvature constraints, the structured inspection route of the inspection drone is automatically planned. The structured inspection route is converted into a recognizable route instruction file and imported into the flight control system of the inspection drone.

[0007] Preferably, the step of extracting inspection image frames from the inspection video stream and performing target detection on the inspection image frames based on a deep learning model to generate multi-scene inspection target points specifically includes: Verify the status of the inspection video stream. If the inspection video stream is in a normal state, submit an asynchronous task through the thread pool to extract the inspection image frame from the inspection video stream. The resolution of the inspection image frames is checked until the preset inspection image resolution is reached; The inspection image frame is input into the deep learning model to perform target detection, generating the multi-scene inspection target points containing target point location, target point type and target detection confidence. A processing timeout mechanism is set up so that if the time taken for the target detection process exceeds a preset processing time threshold, the inspection image frame that has not undergone target detection is returned.

[0008] Preferably, the step of converting the pixel coordinates of the multi-scene inspection target points into actual geographic coordinates based on the camera matrix and digital elevation model of the inspection drone specifically includes: Construct the camera matrix, which includes camera intrinsic parameters, namely distortion coefficients and camera coordinate system; Based on the distortion coefficient, the pixel coordinates of the multi-scene inspection target points are distorted; The pixel coordinates are transformed to the ray direction of the camera coordinate system, and then the ray direction is sequentially transformed to the gimbal coordinate system, the UAV body coordinate system, and the northeast-east coordinate system to generate the actual geographic coordinates. The elevation value of the actual geographic coordinates is then corrected using the digital elevation model.

[0009] Preferably, the step of automatically planning the structured inspection route of the inspection drone based on the actual geographical coordinates of the multi-scenario inspection target points and the path curvature constraint conditions specifically includes: A basic inspection route is generated based on the actual geographic coordinates of the multi-scenario inspection target points, and the basic inspection route includes multiple basic inspection sub-routes composed of basic inspection waypoints. The heading angle of the inspection drone at the basic inspection waypoint is obtained. When the absolute value of the difference between the heading angles of the inspection drone at adjacent basic inspection waypoints exceeds the preset turning angle threshold, the structured inspection route is generated by automatically generating a smooth transition turning sub-route between adjacent basic inspection waypoints to replace the basic inspection sub-route, in combination with the path curvature constraint condition and using a continuous curvature path planning algorithm. Dynamically configure UAV inspection action parameters for the structured inspection waypoints in the structured inspection route.

[0010] Preferably, an improved ant colony optimization algorithm is used to determine the basic inspection waypoints in the basic inspection route, and the corresponding calculation formula is as follows: In the formula, This represents the actual geographic coordinate vector corresponding to the nth basic inspection waypoint in the basic inspection route; This represents the actual geographic coordinate vector corresponding to the kth multi-scenario inspection target point; This represents the coordinate correction vector of the k-th multi-scenario inspection target point; This indicates the total number of basic inspection waypoints in the basic inspection route; This represents the velocity vector of the inspection drone at the nth basic inspection waypoint. The weighting coefficient represents the velocity vector of the inspection drone at the nth basic inspection waypoint; This represents the elevation gradient at the k-th multi-scenario inspection target point. The Euclidean norm; The square of the Euclidean norm represents the distance between the k-th multi-scenario inspection target point and the center coordinate point of the nearest no-fly zone; Indicates the inspection safety distance coefficient; This represents the actual geographic coordinate vector corresponding to the center coordinate point of the nearest no-fly zone.

[0011] Preferably, the heading angle of the inspection drone at the basic inspection waypoint The calculation formula is as follows: In the formula, This represents the actual geographic coordinates of the nth basic inspection waypoint; This represents the actual geographic coordinates of the (n-1)th basic inspection waypoint; This represents the smoothing correction amount of the heading angle of the inspection drone at the nth basic inspection waypoint; Represents the arctangent function in the four quadrants; This represents the smoothing coefficient of the heading angle of the inspection drone; This indicates the elevation influence coefficient of the inspection drone; This represents the heading angle of the inspection drone at the (n-1)th basic inspection waypoint; This represents the heading angle of the inspection drone at the (n-2)th basic inspection waypoint; This represents the constraint value for the maximum heading angle variation of the inspection drone.

[0012] Preferably, the smooth transition turning sub-line between the nth basic inspection waypoint and the (n-1)th basic inspection waypoint is set as follows: , This represents the m-th smooth transition turning sub-waypoint between the n-th basic waypoint and the (n-1)-th basic waypoint, and Corresponding actual geographic coordinate vector The expression is as follows: In the formula, This represents the actual geographic coordinate vector corresponding to the (n-1)th basic inspection waypoint in the basic inspection route; Indicates a smooth transition to a turn point Smooth transition correction vector at the location; Indicates a smooth transition to a turn point The turning normal vector at the location; Indicates a smooth transition to a turn point Curvature at the turning point; This indicates the maximum turning offset distance of the inspection drone; Indicates taking and The smaller value in the range.

[0013] Preferably, the step of converting the structured inspection route into a recognizable route instruction file and importing it into the flight control system of the inspection UAV specifically includes: Obtain a preset route template file containing waypoint placeholders; Extract waypoint information from the structured inspection route and assign the waypoint information to the waypoint placeholders in the preset route template file. Configure corresponding structured flight path instructions for the waypoint placeholders that are assigned the waypoint information, generate the recognizable flight path instruction file, and import it into the flight control system of the inspection UAV for execution.

[0014] A vision-recognition-based unmanned aerial vehicle (UAV) automated inspection route generation system, the system comprising: The video acquisition module is used to acquire inspection video streams of the inspection area through the camera device mounted on the inspection drone; The target detection module is used to extract the inspection image frames of the inspection video stream, and perform target detection on the inspection image frames based on a deep learning model to generate multi-scene inspection target points. The coordinate transformation module is used to convert the pixel coordinates of the multi-scene inspection target points into actual geographic coordinates based on the camera matrix and digital elevation model of the inspection drone. The route planning module is used to automatically plan the structured inspection route of the inspection drone based on the actual geographical coordinates of the multi-scenario inspection target points and the path curvature constraint conditions. The route execution module is used to convert the structured inspection route into a recognizable route instruction file and import it into the flight control system of the inspection UAV.

[0015] Compared with related technologies, the present invention provides a method and system for generating automatic inspection routes for unmanned aerial vehicles based on visual recognition, which has the following advantages: This invention uses a camera device mounted on an inspection drone to collect inspection video streams of the inspection area; extracts inspection image frames from the inspection video stream, and performs target detection on the inspection image frames based on a deep learning model to generate multi-scene inspection target points; based on the camera matrix and digital elevation model of the inspection drone, converts the pixel coordinates of the multi-scene inspection target points into actual geographic coordinates; according to the actual geographic coordinates of the multi-scene inspection target points and combined with path curvature constraints, automatically plans a structured inspection route for the inspection drone; converts the structured inspection route into a recognizable route instruction file and imports it into the flight control system of the inspection drone. Thus, by integrating visual recognition and intelligent route planning technologies, the entire drone inspection process is made efficient, accurate, automated, and intelligent, significantly improving system adaptability, inspection quality, and operational safety.

[0016] This invention automatically collects inspection video streams of the inspection area using a camera device mounted on an inspection drone. Combined with a deep learning model, it detects and identifies inspection target points in multiple scenarios, replacing the traditional method of manually designing flight paths and shooting locations. This significantly reduces manpower and time costs, while avoiding the instability in inspection quality caused by human error. By combining a camera matrix with a digital elevation model, this invention converts the pixel coordinates of inspection target points in multiple scenarios into actual geographic coordinates, providing accurate spatial data support for flight path planning. Based on an improved ant colony optimization algorithm, this invention determines basic inspection waypoints and calculates the heading angle of the inspection drone at these waypoints. Combined with path curvature constraints, a continuous curvature path planning algorithm generates smooth transition turning sub-pathways, effectively solving the problem that traditional drone inspection methods struggle to adapt to a large number of dispersed inspection targets and cannot efficiently plan optimal routes. This ensures that drones can efficiently and safely complete inspection tasks even in complex environments. Furthermore, this invention can dynamically configure UAV inspection action parameters for structured inspection waypoints, making the routes more aligned with actual needs and having a good adaptability effect. It also converts structured inspection routes into recognizable route instruction files and imports them into the flight control system through preset route template files, achieving seamless connection from route planning to execution. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a method for generating automatic inspection routes for unmanned aerial vehicles based on visual recognition, provided in an embodiment of the present invention; Figure 2 A system block diagram of an automatic inspection route generation system for unmanned aerial vehicles based on vision recognition, provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 The diagram shown is a flowchart of a method for generating automatic inspection routes for unmanned aerial vehicles based on visual recognition, provided in an embodiment of the present invention. Figure 1 The execution entity of the method shown can be a software and / or hardware device. The execution entity of this application can include, but is not limited to, at least one of the following: user equipment, network equipment, etc. User equipment can include, but is not limited to, computers, smartphones, personal digital assistants (PDAs), and the aforementioned electronic devices. Network equipment can include, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers. Cloud computing is a type of distributed computing, consisting of a super virtual computer composed of a group of loosely coupled computers. This embodiment does not limit this. Steps S1 to S5 are detailed as follows: S1, collects inspection video streams of the inspection area through the camera device carried by the inspection drone; The inspection drone possesses stable flight capabilities and payload adaptability, enabling it to carry corresponding equipment to complete long-duration, long-distance flight operations according to the needs of the inspection scenario. Furthermore, the drone's flight attitude and position control precision meet the basic requirements for target acquisition and flight path execution. The camera device is a high-definition imaging device adapted to the inspection mission, possessing anti-shake and anti-interference capabilities. Its imaging resolution and frame rate are adapted to target recognition needs under different lighting and distance conditions, and it works in conjunction with the drone's flight control system to ensure continuous and stable image acquisition during the drone's flight. The inspection area refers to the pre-defined spatial range for inspection operations, covering areas containing infrastructure such as power lines, oil pipelines, and factory facilities. The inspection video stream refers to the continuous acquisition and output of dynamic image sequences by the camera device, ensuring continuous footage without significant stuttering or distortion, and fully covering key points and target objects within the inspection area.

[0020] In practical applications, the coordinated use of inspection drones and camera devices replaces the traditional manual inspection method that relies on human observation or fixed equipment to collect information, achieving dynamic and comprehensive coverage of the inspection area.

[0021] S2, extract the inspection image frames from the inspection video stream, and perform target detection on the inspection image frames based on a deep learning model to generate multi-scene inspection target points; The step of extracting inspection image frames from the inspection video stream and performing target detection on the inspection image frames based on a deep learning model to generate multi-scene inspection target points specifically includes: Verify the status of the inspection video stream. If the inspection video stream is in a normal state, submit an asynchronous task through the thread pool to extract the inspection image frame from the inspection video stream. The resolution of the inspection image frames is checked until the preset inspection image resolution is reached; The inspection image frame is input into the deep learning model to perform target detection, generating the multi-scene inspection target points containing target point location, target point type and target detection confidence. A processing timeout mechanism is set up so that if the time taken for the target detection process exceeds a preset processing time threshold, the inspection image frame that has not undergone target detection is returned.

[0022] By verifying the status of the inspection video stream, it is possible to identify and confirm that there are no data interruptions or severe image distortions in the inspection video stream. Only inspection video streams in a normal state can provide a reliable source of raw data for the extraction of inspection image frames, thus avoiding deviations in subsequent target detection results due to abnormal source data.

[0023] By submitting asynchronous tasks through a thread pool to extract image frames, parallel processing of inspection image frame extraction and continuous monitoring of inspection video stream status can be achieved, reducing waiting time in each stage. This is especially suitable for the high-efficiency processing requirements of continuous video streams in large-scale inspection scenarios.

[0024] The preset inspection image resolution can be determined by considering the actual characteristics of the inspection target, the inspection distance, and the input standards of the deep learning model. Repeated verification ensures that the inspection image frames clearly present the detailed features of the inspection target, preventing insufficient resolution from causing the deep learning model to fail to accurately identify the target and affecting the accuracy of subsequent multi-scenario inspection target point generation. For multi-scenario inspection target points, the target point location provides spatial coordinates for subsequent route planning, the target point type clarifies the key inspection objects, and the target detection confidence score ensures the reliability of the target recognition results.

[0025] In addition, by handling timeout mechanisms, the model computation delay problem in complex inspection scenarios can be addressed. The preset processing time threshold can be set comprehensively based on the actual inspection efficiency requirements and the model computation complexity. When the target detection time exceeds the threshold, the image frames that have not been detected are returned in a timely manner, which can prevent the system from being blocked for a long time. This facilitates the separate processing of the inspection image frames in the future, ensuring the continuity and stability of the entire inspection process, and balancing detection efficiency and overall work progress.

[0026] For example, in a power line inspection scenario, an inspection drone equipped with a suitable high-definition camera takes off and inspects a high-voltage transmission line in a certain area, collecting inspection video streams in real time. First, the status of the inspection video stream is verified by monitoring data transmission stability and checking for severe distortion or interruptions in the image. Once it is confirmed to be in a normal state, an asynchronous task is submitted using a thread pool. While continuously monitoring the inspection video stream status, the extraction of inspection image frames is completed in parallel, avoiding discontinuous image frame acquisition due to single-task blocking. After extracting the inspection image frames, image resolution is checked for targets such as insulators, conductors, and tower hardware in the power line until the preset inspection image resolution standard is met, ensuring that target details can be accurately captured. The image frames are then input into a deep learning model, which quickly performs target detection, generates inspection target points for multiple scenes, marks the specific location of the target points, clarifies the defect type of "insulator damage," and the corresponding target detection confidence level.

[0027] S3, based on the camera matrix and digital elevation model of the inspection drone, convert the pixel coordinates of the multi-scene inspection target points into actual geographic coordinates; The conversion of pixel coordinates of the multi-scene inspection target points into actual geographic coordinates based on the camera matrix and digital elevation model of the inspection drone specifically includes: Construct the camera matrix, which includes camera intrinsic parameters, namely distortion coefficients and camera coordinate system; Based on the distortion coefficient, the pixel coordinates of the multi-scene inspection target points are distorted; The pixel coordinates are transformed to the ray direction of the camera coordinate system, and then the ray direction is sequentially transformed to the gimbal coordinate system, the UAV body coordinate system, and the northeast-east coordinate system to generate the actual geographic coordinates. The elevation value of the actual geographic coordinates is then corrected using the digital elevation model.

[0028] The camera matrix includes camera intrinsic parameters, namely distortion coefficients and the camera coordinate system. Distortion coefficients correct pixel coordinate distortion caused by the optical characteristics of the camera lens. The camera coordinate system is a three-dimensional coordinate system established with the camera's optical center as the origin. Then, based on the distortion coefficients, error correction can be performed on the pixel coordinates of target points in multiple scenes, eliminating coordinate deviations caused by lens optical distortion. Pixel coordinates are the two-dimensional coordinates of the target point on the inspection image frame. The ray direction is the spatial direction vector from the camera's optical center to the target point, formed after transforming the pixel coordinates from the two-dimensional image space to the camera coordinate system. The gimbal coordinate system has the gimbal rotation center as its origin, adapting to the motion characteristics of the gimbal driving the camera device's rotation. The UAV body coordinate system has the inspection UAV's fuselage center as its origin and is related to the UAV's flight attitude. The NE-G coordinate system is a geographic coordinate system with due north as the X-axis, due east as the Y-axis, and vertical upwards as the Z-axis; it is a standard coordinate system representing the actual geographical location. These three elements sequentially transform the ray direction, ultimately mapping the target point's location to geographic space, forming actual geographic coordinates containing longitude, latitude, and elevation values. The digital elevation model (DEM) is a geographic information model containing terrain elevation data of the inspection area. By correcting the elevation values ​​of the actual geographic coordinates, the influence of terrain undulations on the target point's elevation positioning can be eliminated.

[0029] When transforming pixel coordinates to the camera coordinate system ray direction, the spatial direction of the target point relative to the camera is first determined. Then, it is sequentially transformed to the gimbal coordinate system and the UAV body coordinate system, gradually associating the target point position with the gimbal rotation state and body attitude of the inspection UAV. Finally, it is transformed to the NE-G coordinate system, incorporating the local spatial coordinates into a global coordinate system corresponding to the actual geographical environment, thereby generating coordinate data with actual geographic positioning significance. Then, the elevation value is corrected through a digital elevation model to ensure that the elevation parameters of the actual geographic coordinates are consistent with the actual terrain height, ensuring that the inspection UAV can adapt to terrain undulations and maintain a safe flight altitude.

[0030] S4. Based on the actual geographical coordinates of the multi-scenario inspection target points and combined with the path curvature constraint, automatically plan the structured inspection route of the inspection drone. The automatic planning of a structured inspection route for the inspection drone based on the actual geographical coordinates of the multi-scenario inspection target points and path curvature constraints specifically includes: A basic inspection route is generated based on the actual geographic coordinates of the multi-scenario inspection target points, and the basic inspection route includes multiple basic inspection sub-routes composed of basic inspection waypoints. The heading angle of the inspection drone at the basic inspection waypoint is obtained. When the absolute value of the difference between the heading angles of the inspection drone at adjacent basic inspection waypoints exceeds the preset turning angle threshold, the structured inspection route is generated by automatically generating a smooth transition turning sub-route between adjacent basic inspection waypoints to replace the basic inspection sub-route, in combination with the path curvature constraint condition and using a continuous curvature path planning algorithm. Dynamically configure UAV inspection action parameters for the structured inspection waypoints in the structured inspection route.

[0031] The basic inspection route consists of multiple basic inspection sub-routes, and each basic inspection sub-routes is formed by connecting two adjacent basic inspection points.

[0032] The heading angle is the flight direction angle of the inspection drone at the basic inspection waypoints. It is calculated by combining the actual geographical coordinates of adjacent basic inspection waypoints, and a smoothing correction is introduced to eliminate abrupt changes in heading to ensure the consistency of the inspection drone's flight direction. The preset turning angle threshold is a standard value for determining whether the basic inspection sub-line is suitable for the flight performance of the inspection drone. When the absolute value of the difference in heading angle between adjacent basic inspection waypoints exceeds this threshold, it indicates that the turning amplitude of the original basic inspection sub-line is too large, which may affect the flight stability of the inspection drone, and the route optimization process needs to be initiated.

[0033] The path curvature constraint is set based on parameters such as the maneuverability and load stability of the inspection UAV. It is used to constrain the curvature changes of the flight path turning, avoiding sharp or steep turns that exceed the maneuverability of the inspection UAV, thus ensuring flight safety and the stability of the inspection actions. Under the premise of satisfying the path curvature constraint, a smooth transition turning sub-path with continuously changing curvature is generated through a continuous curvature path planning algorithm, replacing the original basic inspection sub-path with excessive turning amplitude.

[0034] The smooth transition turning sub-path has continuous curvature changes that conform to path curvature constraints, enabling smooth turning between adjacent basic inspection waypoints. This effectively improves the stability and safety of the inspection UAV flight while ensuring the clarity of the inspection images. The structured inspection path is formed by integrating the smooth transition turning sub-path with the optimized basic inspection sub-path. It possesses the dual characteristics of basic coverage and smooth turning, ensuring coverage of all multi-scenario inspection target points while conforming to the flight performance of the inspection UAV, thus achieving both high efficiency and safety.

[0035] The drone inspection action parameters are execution parameters that are dynamically configured according to the actual inspection needs of the structured inspection waypoints. They cover key commands such as the hovering time of the inspection drone at the waypoint, the shooting angle of the camera device, and the flight speed, which makes the drone's actions adaptable in different inspection scenarios and further improves the quality and efficiency of inspection.

[0036] An improved ant colony optimization algorithm is used to determine the basic inspection waypoints in the basic inspection route. The corresponding calculation formula is as follows: In the formula, This represents the actual geographic coordinate vector corresponding to the nth basic inspection waypoint in the basic inspection route; This represents the actual geographic coordinate vector corresponding to the kth multi-scenario inspection target point; This represents the coordinate correction vector of the k-th multi-scenario inspection target point; This indicates the total number of basic inspection waypoints in the basic inspection route; This represents the velocity vector of the inspection drone at the nth basic inspection waypoint. The weighting coefficient represents the velocity vector of the inspection drone at the nth basic inspection waypoint; This represents the elevation gradient at the k-th multi-scenario inspection target point. The Euclidean norm; The square of the Euclidean norm represents the distance between the k-th multi-scenario inspection target point and the center coordinate point of the nearest no-fly zone; Indicates the inspection safety distance coefficient; This represents the actual geographic coordinate vector corresponding to the center coordinate point of the nearest no-fly zone.

[0037] Among them, the coordinate correction vector is used to eliminate the slight deviations that may exist in the actual geographic coordinates of the inspection target points in the early stage of the conversion process, such as the cumulative error when converting pixel coordinates to geographic coordinates. Through targeted correction, the spatial position of the basic inspection waypoints and the target points can be accurately aligned, avoiding the waypoints from deviating from the inspection focus due to coordinate deviations.

[0038] The weighting coefficient of the inspection drone's velocity vector is not a fixed value, but is dynamically adjusted based on the distance between adjacent basic inspection waypoints and the inspection priority of the inspection scenario. For example, a slower flight velocity vector can be set for basic inspection waypoints in high-priority inspection scenarios to allow sufficient time for detailed inspection; while a faster velocity vector is adapted for basic inspection waypoints in ordinary inspection scenarios to ensure overall inspection efficiency.

[0039] Understandably, when using the improved ant colony optimization algorithm to determine basic inspection waypoints, the actual geographic coordinate vector of each basic inspection waypoint is calculated. Based on the actual geographic coordinates of the multi-scenario inspection target points, and combined with their coordinate correction vectors, the algorithm ensures a precise correspondence between the basic inspection waypoints and the multi-scenario inspection target points. Simultaneously, the velocity vector of the inspection drone at the basic inspection waypoint and its corresponding weight coefficient are incorporated to adapt to the drone's flight status. The Euclidean norm of the elevation gradient of the multi-scenario inspection target points is introduced to adapt to terrain undulations. The square of the Euclidean norm of the distance between the target point and the nearest no-fly zone center, the inspection safety distance coefficient, and the coordinate vector of the no-fly zone center are combined to mitigate no-fly zone risks. Finally, through comprehensive calculation of multiple factors, the optimal basic inspection waypoint is determined, ensuring the practicality and safety of the basic inspection routes.

[0040] The heading angle of the inspection drone at the basic inspection waypoint The calculation formula is as follows: In the formula, This represents the actual geographic coordinates of the nth basic inspection waypoint; This represents the actual geographic coordinates of the (n-1)th basic inspection waypoint; This represents the smoothing correction amount of the heading angle of the inspection drone at the nth basic inspection waypoint; Represents the arctangent function in the four quadrants; This represents the smoothing coefficient of the heading angle of the inspection drone; This indicates the elevation influence coefficient of the inspection drone; This represents the heading angle of the inspection drone at the (n-1)th basic inspection waypoint; This represents the heading angle of the inspection drone at the (n-2)th basic inspection waypoint; This represents the constraint value for the maximum heading angle variation of the inspection drone.

[0041] One approach is to use the four-quadrant arctangent function to calculate the initial direction angle corresponding to the line connecting adjacent basic inspection waypoints based on their actual geographical coordinates. This allows for accurate differentiation of headings in different directions and avoids directional deviations caused by calculations in a single quadrant.

[0042] Regarding the heading angle smoothing correction, the heading angle smoothing coefficient is set according to the flight stability requirements of the inspection drone. This allows for buffering and adjusting the heading angle changes between adjacent basic inspection waypoints, preventing the inspection drone from turning sharply and becoming unstable due to excessive differences in heading angles between basic inspection waypoints, and ensuring continuous and smooth heading switching.

[0043] The elevation influence coefficient is used to adapt to the terrain undulations of the inspection area. When the elevation value of the location of the basic inspection waypoint changes, this coefficient can adjust the heading angle calculation result accordingly to offset the interference of terrain elevation difference on the heading stability of the inspection drone and ensure the accuracy of the heading angle under complex terrain.

[0044] The maximum heading angle variation constraint can prevent drastic changes in heading angle between adjacent basic inspection waypoints that exceed the controllable range of the inspection drone, thereby avoiding flight safety risks caused by over-turning.

[0045] By using the above method, the calculation of the heading angle of the current waypoint not only depends on the directly adjacent basic inspection waypoints, but also fits the heading trend of the entire basic inspection route, further improving the consistency and rationality of the heading angle.

[0046] Define the smooth transition turning sub-path between the nth basic inspection waypoint and the (n-1)th basic inspection waypoint as follows: , This represents the m-th smooth transition turning sub-waypoint between the n-th basic waypoint and the (n-1)-th basic waypoint, and Corresponding actual geographic coordinate vector The expression is as follows: In the formula, This represents the actual geographic coordinate vector corresponding to the (n-1)th basic inspection waypoint in the basic inspection route; Indicates a smooth transition to a turn point Smooth transition correction vector at the location; Indicates a smooth transition to a turn point The turning normal vector at the location; Indicates a smooth transition to a turn point Curvature at the turning point; This indicates the maximum turning offset distance of the inspection drone; Indicates taking and The smaller value in the range.

[0047] When determining the actual geographic coordinate vector of the m-th smooth transition turning sub-waypoint between the n-th basic inspection waypoint and the (n-1)-th basic inspection waypoint, the actual geographic coordinate vector of the (n-1)-th basic inspection waypoint is used as the reference starting point to provide an initial spatial reference for the entire smooth turning path. This ensures that the smooth transition turning sub-line can accurately connect the (n-1)-th and n-th basic inspection waypoints, avoiding any disconnection between the turning path and the original basic inspection line.

[0048] The smooth transition correction vector is used to eliminate path deviations that may occur during the turning process, such as waypoint offsets caused by slight fluctuations in the flight attitude of the inspection drone or terrain interference. Through targeted correction, the actual geographic coordinates of each smooth transition turning sub-waypoint are made to better fit the preset turning trajectory, ensuring the accuracy of the turning path.

[0049] The steering normal vector is used to define the directional reference for a smooth transition steering. Its direction setting conforms to the trend of heading angle change between adjacent basic inspection waypoints, ensuring that the inspection UAV can fly smoothly along the preset steering direction and avoid flight path disorder caused by steering direction deviation.

[0050] The turning curvature directly determines the curvature of the smooth transition turning sub-path. It is set comprehensively based on the difference in heading angle between adjacent basic inspection waypoints and the flight performance of the inspection drone. By controlling the turning curvature, the turning path presents a continuous and smooth curve shape, preventing the inspection drone from turning sharply and becoming unstable due to excessive turning curvature.

[0051] The maximum turning offset distance is used to limit the maximum deviation range of the smooth transition turning sub-waypoint relative to the basic inspection route, preventing the inspection drone from deviating from the inspection area or entering no-fly or dangerous areas during the turning process. By taking... and The smaller value in the value can define a safety boundary for the turning path, preventing the inspection drone from deviating from the inspection area or entering a dangerous area when turning, thus ensuring both smooth turning and flight safety.

[0052] The above method enables the smooth transition turning sub-path to not only fill the gap in the heading connection between adjacent basic inspection waypoints, but also to ensure the stability and safety of the inspection drone's turning process.

[0053] S5, convert the structured inspection route into a recognizable route instruction file and import it into the flight control system of the inspection drone.

[0054] The step of converting the structured inspection route into a recognizable route instruction file and importing it into the flight control system of the inspection drone specifically includes: Obtain a preset route template file containing waypoint placeholders; Extract waypoint information from the structured inspection route and assign the waypoint information to the waypoint placeholders in the preset route template file. Configure corresponding structured flight path instructions for the waypoint placeholders that are assigned the waypoint information, generate the recognizable flight path instruction file, and import it into the flight control system of the inspection UAV for execution.

[0055] The preset route template file is a standardized document framework pre-defined to adapt to the flight control system of the inspection drone. It provides a unified command format and data layout to avoid unrecognizable route commands due to compatibility issues with the flight control system. Waypoint placeholders, as blank data positions in the preset route template file, are set sequentially according to the waypoint order of the structured inspection route. They reserve fixed positions for the accurate filling of subsequent waypoint information, ensuring the compatibility of waypoint data with the preset route template file format.

[0056] The waypoint information of structured inspection waypoints includes the actual geographic coordinates of the waypoint, the heading angle of the inspection drone at the waypoint, and the inspection action parameters. By assigning the waypoint information to waypoint placeholders, the previously planned structured inspection route data can be transformed into a standardized data format that can be recognized by the preset route template file.

[0057] Structured flight path instructions are specific execution commands generated based on the filled waypoint information. Combining the instruction protocol of the inspection UAV flight control system, structured flight path instructions are configured for waypoint placeholders with allocated waypoint information. This transforms the waypoint information into operation instructions that the flight control system can parse, ensuring that the inspection needs of each waypoint can be accurately achieved through operation instructions, ultimately generating a recognizable flight path instruction file.

[0058] After the identifiable route instruction file is imported into the flight control system, the flight control system can automatically control the inspection drone to fly along the structured inspection route according to the waypoint sequence and action requirements in the identifiable route instruction file, and complete the inspection tasks of each waypoint. The whole process realizes the standardization and automation of route planning to actual flight, ensuring the smooth execution of the inspection process.

[0059] Taking urban power distribution network line inspection as an example, the first step is to retrieve the preset route template file adapted to the inspection drone flight control system. This preset route template file contains multiple waypoint placeholders set in the order of the inspection process. Each waypoint placeholder corresponds to a key node in the power distribution network inspection, such as transformers, branch line joints, insulator groups, etc., to ensure that the preset route template file format is fully matched with the flight control system instruction protocol.

[0060] Subsequently, the waypoint information of all structured inspection waypoints in the pre-planned structured inspection route is extracted, including the actual geographical coordinates of each structured inspection waypoint, the heading angle of the inspection drone at the structured inspection waypoint, the preset hovering time and lens shooting parameters, etc. This information is then filled into the waypoint placeholders in the preset route template file one by one according to the order of the structured inspection waypoints. For example, the coordinates and motion parameters of the "transformer inspection waypoint" are filled into the waypoint placeholder with the corresponding number to avoid information misalignment.

[0061] After the population is complete, the corresponding structured flight path instructions are automatically configured based on the waypoint information in each waypoint placeholder. For example, for waypoints for hovering inspection, instructions such as "maintain hovering after reaching the coordinates" and "start high-definition camera shooting" are generated. After all instructions are configured, a recognizable flight path instruction file that can be directly read by the flight control system is generated and imported into the flight control system.

[0062] Before the inspection drone takes off, staff only need to check the key waypoint information in the identifiable route instruction file. Once confirmed, the drone is started, and the flight control system will control the drone to complete the inspection tasks of each structured inspection waypoint in sequence along the structured inspection route according to the order in the identifiable route instruction file. No manual real-time operation or adjustment is required, which greatly improves the efficiency of urban power distribution network inspection and ensures that the tasks of each inspection node are executed accurately.

[0063] like Figure 2 The diagram shown is a system block diagram of a vision-recognition-based unmanned aerial vehicle (UAV) automatic inspection route generation system according to an embodiment of the present invention. The system includes: The video acquisition module is used to acquire inspection video streams of the inspection area through the camera device mounted on the inspection drone; The target detection module is used to extract the inspection image frames of the inspection video stream, and perform target detection on the inspection image frames based on a deep learning model to generate multi-scene inspection target points. The coordinate transformation module is used to convert the pixel coordinates of the multi-scene inspection target points into actual geographic coordinates based on the camera matrix and digital elevation model of the inspection drone. The route planning module is used to automatically plan the structured inspection route of the inspection drone based on the actual geographical coordinates of the multi-scenario inspection target points and the path curvature constraint conditions. The route execution module is used to convert the structured inspection route into a recognizable route instruction file and import it into the flight control system of the inspection UAV.

[0064] Figure 2 The apparatus of the illustrated embodiment can be used to perform corresponding actions. Figure 1 The steps in the method embodiments shown are implemented in a similar manner and have similar technical effects, and will not be repeated here.

[0065] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the steps of a vision recognition-based unmanned aerial vehicle (UAV) automatic inspection route generation method as described above.

[0066] like Figure 3The diagram shown is a hardware structure schematic of an electronic device according to an embodiment of the present invention. The electronic device 30 includes: a processor 31, a memory 32, and a computer program; wherein... The memory 32 is used to store the computer program, and the memory may also be flash memory. The computer program is, for example, an application program or functional module that implements the above method.

[0067] Processor 31 is configured to execute the computer program stored in the memory to implement the various steps performed by the device in the above method. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0068] Alternatively, the memory 32 can be either standalone or integrated with the processor 31.

[0069] When the memory 32 is a device independent of the processor 31, the device may further include: Bus 33 is used to connect the memory 32 and the processor 31.

[0070] A readable storage medium storing a computer program, which, when executed by a processor, is used to implement the steps of a vision recognition-based unmanned aerial vehicle (UAV) automatic inspection route generation method as described above.

[0071] The readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of computer programs from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application-Specific Integrated Circuit (ASIC). Alternatively, the ASIC can be located in a user equipment. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0072] The present invention also provides a program product including executable instructions stored in a readable storage medium. At least one processor of the device can read the executable instructions from the readable storage medium, and the at least one processor executes the executable instructions to cause the device to implement the methods provided in the various embodiments described above.

[0073] In the embodiments of the above-described device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0074] Through the above embodiments, this invention acquires inspection video streams of the inspection area using a camera device mounted on an inspection drone; extracts inspection image frames from the inspection video streams, and performs target detection on the inspection image frames based on a deep learning model to generate multi-scene inspection target points; based on the camera matrix and digital elevation model of the inspection drone, converts the pixel coordinates of the multi-scene inspection target points into actual geographic coordinates; according to the actual geographic coordinates of the multi-scene inspection target points, combined with path curvature constraints, automatically plans a structured inspection route for the inspection drone; converts the structured inspection route into a recognizable route instruction file and imports it into the flight control system of the inspection drone. Thus, by integrating visual recognition and intelligent route planning technologies, the entire drone inspection process is made efficient, accurate, automated, and intelligent, significantly improving system adaptability, inspection quality, and operational safety.

[0075] This invention automatically collects inspection video streams of the inspection area using a camera device mounted on an inspection drone. Combined with a deep learning model, it detects and identifies inspection target points in multiple scenarios, replacing the traditional method of manually designing flight paths and shooting locations. This significantly reduces manpower and time costs, while avoiding the instability in inspection quality caused by human error. By combining a camera matrix with a digital elevation model, this invention converts the pixel coordinates of inspection target points in multiple scenarios into actual geographic coordinates, providing accurate spatial data support for flight path planning. Based on an improved ant colony optimization algorithm, this invention determines basic inspection waypoints and calculates the heading angle of the inspection drone at these waypoints. Combined with path curvature constraints, a continuous curvature path planning algorithm generates smooth transition turning sub-pathways, effectively solving the problem that traditional drone inspection methods struggle to adapt to a large number of dispersed inspection targets and cannot efficiently plan optimal routes. This ensures that drones can efficiently and safely complete inspection tasks even in complex environments. Furthermore, this invention can dynamically configure UAV inspection action parameters for structured inspection waypoints, making the routes more aligned with actual needs and having a good adaptability effect. It also converts structured inspection routes into recognizable route instruction files and imports them into the flight control system through preset route template files, achieving seamless connection from route planning to execution.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating an automatic inspection route of a UAV based on visual recognition, characterized in that, The method comprises: collecting a patrol video stream of a patrol area by a camera device carried by a patrol unmanned aerial vehicle; extracting a patrol image frame of the patrol video stream, and performing target detection on the patrol image frame based on a deep learning model to generate multi-scene patrol target points; converting pixel coordinates of the multi-scene patrol target points into actual geographic coordinates based on a camera matrix of the patrol unmanned aerial vehicle and a digital elevation model; automatically planning a structured patrol route of the patrol unmanned aerial vehicle according to the actual geographic coordinates of the multi-scene patrol target points in combination with a path curvature constraint condition; converting the structured patrol route into an identifiable route instruction file and importing the identifiable route instruction file into a flight control system of the patrol unmanned aerial vehicle. 2.The method of claim 1, wherein, The extracting of the patrol image frame of the patrol video stream and the performing of target detection on the patrol image frame based on the deep learning model to generate the multi-scene patrol target points specifically comprises: verifying a state of the patrol video stream, and if the patrol video stream is in a normal state, submitting an asynchronous task through a thread pool to extract the patrol image frame from the patrol video stream; performing resolution verification on the patrol image frame until a preset patrol image resolution is reached; inputting the patrol image frame into the deep learning model to perform target detection, and generating the multi-scene patrol target points containing target point positions, target point types and target detection confidence levels; setting a processing timeout mechanism, and if a time consumption of the target detection process exceeds a preset processing time threshold, returning the patrol image frame without target detection. 3.The visual recognition based unmanned aerial vehicle automatic inspection route generation method of claim 1, wherein, The converting of the pixel coordinates of the multi-scene patrol target points into the actual geographic coordinates based on the camera matrix of the patrol unmanned aerial vehicle and the digital elevation model specifically comprises: constructing the camera matrix, and the camera matrix contains camera intrinsic parameters, i.e. distortion coefficients and a camera coordinate system; performing distortion correction on the pixel coordinates of the multi-scene patrol target points based on the distortion coefficients; converting the pixel coordinates to a ray direction of the camera coordinate system, and then sequentially converting the ray direction to a gimbal coordinate system, a unmanned aerial vehicle body coordinate system and a north-east-ground coordinate system to generate the actual geographic coordinates, and correcting an elevation value of the actual geographic coordinates through the digital elevation model. 4.The method of claim 1, wherein, The automatically planning of the structured patrol route of the patrol unmanned aerial vehicle according to the actual geographic coordinates of the multi-scene patrol target points in combination with the path curvature constraint condition specifically comprises: generating a basic patrol route based on the actual geographic coordinates of the multi-scene patrol target points, and the basic patrol route comprises a plurality of basic patrol sub-routes composed of basic patrol waypoints; obtaining a heading angle of the patrol unmanned aerial vehicle at the basic patrol waypoints, and when detecting that an absolute value of a difference between the heading angles of the patrol unmanned aerial vehicle at adjacent basic patrol waypoints exceeds a preset turning angle threshold, generating a smooth transition turning sub-route between the adjacent basic patrol waypoints in combination with the path curvature constraint condition by using a continuous curvature path planning algorithm to replace the basic patrol sub-route, and generating the structured patrol route. The unmanned aerial vehicle inspection action parameters are dynamically configured for the structured inspection waypoints in the structured inspection route. 5.The method of claim 4, wherein, The improved ant colony optimization algorithm is adopted to determine the basic inspection waypoints in the basic inspection route, and the corresponding calculation formula is as follows: wherein, denotes the actual geographic coordinate vector corresponding to the nth basic inspection waypoint in the basic inspection route; denotes the actual geographic coordinate vector corresponding to the kth multi-scene inspection target point; denotes the coordinate correction vector of the kth multi-scene inspection target point; denotes the total number of basic inspection waypoints in the basic inspection route; denotes the speed vector of the inspection UAV at the nth basic inspection waypoint; denotes the weight coefficient of the speed vector of the inspection UAV at the nth basic inspection waypoint; denotes the Euclidean norm of the elevation gradient at the kth multi-scene inspection target point ; denotes the square of the Euclidean norm of the distance between the kth multi-scene inspection target point and the center coordinate point of the nearest no-fly zone; denotes the inspection safety distance coefficient; denotes the actual geographic coordinate vector corresponding to the center coordinate point of the nearest no-fly zone. 6.The method of claim 4, wherein, The heading angle of the inspection unmanned aerial vehicle at the basic inspection waypoint The calculation formula is as follows: wherein, represents the actual geographic coordinates of the nth basic inspection waypoint; represents the actual geographic coordinates of the (n-1)th basic inspection waypoint; represents the heading angle smoothing correction value of the inspection UAV at the nth basic inspection waypoint; represents the four-quadrant arctangent function; represents the heading angle smoothing coefficient of the inspection UAV; represents the elevation influence coefficient of the inspection UAV; represents the heading angle of the inspection UAV at the (n-1)th basic inspection waypoint; represents the heading angle of the inspection UAV at the (n-2)th basic inspection waypoint; represents the maximum heading angle change constraint value of the inspection UAV. 7.The method of claim 4, wherein, The smooth transition steering sub-segment between the nth basic inspection waypoint and the (n-1)th basic inspection waypoint is set as , The mth smooth transition steering sub-waypoint between the nth basic inspection waypoint and the (n-1)th basic inspection waypoint is represented as Pm(n-1), and The corresponding actual geographic coordinate vector The expression is as follows: wherein, denotes the actual geographic coordinate vector corresponding to the n-1th basic inspection waypoint in the basic inspection route; denotes the smooth transition correction vector at the smooth transition steering sub-waypoint ; denotes the steering normal vector at the smooth transition steering sub-waypoint ; denotes the steering curvature at the smooth transition steering sub-waypoint ; denotes the maximum steering deviation distance of the inspection unmanned aerial vehicle; denotes the smaller value in and . 8.The method of claim 1, wherein, The structured inspection route is converted into an identifiable route instruction file and imported into the flight control system of the inspection unmanned aerial vehicle, and specifically includes: A preset route template file containing waypoint placeholders is acquired. Waypoint information of the structured inspection waypoints in the structured inspection route is extracted, and the waypoint information is correspondingly assigned to the waypoint placeholders in the preset route template file. The corresponding structured route instructions are configured for the waypoint placeholders to which the waypoint information is assigned, the identifiable route instruction file is generated, and the identifiable route instruction file is imported into the flight control system of the inspection unmanned aerial vehicle for execution.

9. A visual recognition-based unmanned aerial vehicle automatic inspection route generation system, applied to the visual recognition-based unmanned aerial vehicle automatic inspection route generation method according to any one of claims 1-8, characterized in that, The system comprises: A video acquisition module is configured to acquire inspection video streams of an inspection area through a camera device carried by an inspection unmanned aerial vehicle. A target detection module is configured to extract inspection image frames of the inspection video streams and perform target detection on the inspection image frames based on a deep learning model to generate multi-scene inspection target points. A coordinate conversion module is configured to convert pixel coordinates of the multi-scene inspection target points into actual geographic coordinates based on a camera matrix of the inspection unmanned aerial vehicle and a digital elevation model. A route planning module is configured to automatically plan a structured inspection route of the inspection unmanned aerial vehicle according to the actual geographic coordinates of the multi-scene inspection target points in combination with a path curvature constraint condition. A route execution module is configured to convert the structured inspection route into an identifiable route instruction file and import the identifiable route instruction file into a flight control system of the inspection unmanned aerial vehicle.

Images