Substation unmanned aerial vehicle inspection method and system based on visual navigation

Abstract

The application discloses a kind of based on visual navigation substation unmanned aerial vehicle inspection method and system, comprising: based on the video stream data of unmanned aerial vehicle acquisition, obtain multiple continuous video frame images;Video frame image is input into feature extraction network, and the substation road environment feature is extracted, and the extracted feature is respectively input into yaw angle prediction network and horizontal flight prediction network, and the yaw angle parameter of each direction substation road and the translation parameter required for unmanned aerial vehicle flight are output;According to the parameter information output, based on Gaussian mixture model, determine the yaw angle of unmanned aerial vehicle, and then generate unmanned aerial vehicle rotation direction control instruction and generate unmanned aerial vehicle translation control instruction;According to control instruction, control unmanned aerial vehicle flight, and carry out inspection.The application learns through the visual image of substation, so that unmanned aerial vehicle can recognize the pre-set inspection path and surrounding environmental obstacles, get rid of the dependence on GPS navigation signal, realize substation unmanned aerial vehicle automatic inspection.

Description

Technical Field

[0001] This invention relates to the field of substation inspection technology, and in particular to a visual navigation-based unmanned aerial vehicle (UAV) inspection method and system for substations. Background Technology

[0002] Remote video surveillance systems are widely used in substation inspections, and 220kV and below substations have largely achieved unmanned operation. However, current remote video surveillance systems are typically simple video monitoring loops, lacking autonomous intelligent analysis of the content contained in video and environmental data. Safety monitoring still mainly relies on the human observation of operators, resulting in high labor intensity, low automation and operational efficiency. Furthermore, the complex wiring at inspection sites poses a threat to the personal safety of personnel. The effectiveness of substation monitoring is affected by various factors such as the quality and technical level of inspection personnel and the environment, making it difficult to guarantee the reliability of substation operation under complex conditions.

[0003] In recent years, with the development of next-generation artificial intelligence technologies such as deep learning and neuromorphic computing, autonomous substation inspection methods such as fixed cameras and robots have emerged, gradually replacing traditional manual inspections. Fixed camera inspections are limited by the complex space and numerous obstructions in substations, requiring the deployment of a large number of intelligent cameras and resulting in significant maintenance and repair workload. Inspection robots are suitable for open outdoor areas, but their range of motion is limited by road surface unevenness, and their field of view is narrow, making it difficult to monitor the entire substation area. In contrast, drones offer advantages such as a wide field of view, lightweight flexibility, high efficiency, convenience, and low cost in substation inspections, and have been widely used in transmission line inspections in recent years.

[0004] Currently, drone inspections primarily rely on GPS navigation, with manual pre-setting of flight routes to guide the drones. However, due to building obstructions and electromagnetic interference, the GPS signal quality at substations is poor, making it difficult to provide accurate location information and posing significant challenges to autonomous drone inspections. Furthermore, since most drones are currently semi-open platforms, it is impossible to directly read drone status information and video stream data from gimbal cameras. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, this invention provides a visual navigation-based method and system for substation drone inspection. By utilizing APKs (Android application packages) commonly supported by drone manufacturers to read and process drone status information and camera video stream data, and through visual image learning from the substation, the drone can recognize pre-set inspection paths and surrounding environmental obstacles. This eliminates reliance on GPS navigation signals, solves the problem of drones being affected by the quality of GPS signals at substations during inspections, and enables automatic drone inspection of substations, ensuring the safe operation of "unmanned" substations.

[0006] Firstly, this disclosure provides a method for unmanned aerial vehicle (UAV) inspection of substations based on visual navigation.

[0007] A visual navigation-based unmanned aerial vehicle (UAV) inspection method for substations includes:

[0008] The system acquires video stream data collected by the drone, preprocesses the video stream data, and obtains multiple consecutive video frame images.

[0009] The video frame image is input into the feature extraction network to extract the substation road environment features. The substation road environment features are then input into the yaw angle prediction network and the horizontal flight prediction network, respectively, and the yaw angle parameters of the substation road in each direction and the translation parameters required for UAV flight are output.

[0010] Based on the deflection angle parameters of the substation roads in each direction, and using a Gaussian mixture model, the yaw angle of the UAV is determined, and then the rotation direction control command of the UAV is generated; based on the translation parameters required for the flight of the UAV, the translation control command of the UAV is generated.

[0011] The drone is controlled to fly and perform inspections based on the drone's rotation direction control command and the drone's translation control command.

[0012] A further technical solution involves a yaw angle prediction network consisting of a ResNet residual block and four fully connected layers. For each input feature, the yaw angle prediction network outputs yaw angle parameters of the substation road in n directions. and

[0013] A further technical solution is that the loss function of the yaw angle prediction network is the standard negative log-likelihood function, as shown in the formula:

[0014]

[0015] In the above formula, D is the direction label of the training dataset, j represents the j-th sample, j∈[1[N], N is the total number of samples in the training dataset, and x∈(-1,1) represents the range of road deflection angles in the image. The normalized value, P, represents the probability that the deflection angle is in the direction of the road. The output of the yaw angle prediction network is the yaw angle parameter of each substation road output by the yaw angle prediction network; n represents the number of road directions of the substation roads.

[0016] A further technical solution involves a horizontal flight prediction network comprising a ResNet residual block and three fully connected layers, used to predict the translational parameters required for UAV flight.

[0017] In a further technical solution, the loss function of the horizontal flight prediction network is the mean squared error loss, and the formula is:

[0018]

[0019] In the above formula, T represents the translation label. To predict the translation label, i represents the i-th sample, i∈[1[N], and N is the total number of samples in the training dataset.

[0020] A further technical solution involves determining the yaw angle of the UAV based on the deflection angle parameters of the substation road in each direction, using a Gaussian mixture model, and then generating UAV rotation direction control commands, including:

[0021] Based on the yaw angle parameters of the substation roads in each direction output by the yaw angle prediction network, the probability distribution of each substation road direction is obtained based on the Gaussian mixture model.

[0022] The midpoint of the probability distribution segment corresponding to the part with a probability greater than a predetermined threshold is taken as the candidate rotation direction for UAV flight.

[0023] The highest probability candidate rotation direction of the UAV flight is selected as the output rotation direction, and the output rotation direction is used as the UAV yaw angle;

[0024] The rotation direction is linearly converted into the UAV's rotation angular velocity by linear scaling, thereby generating the UAV's rotation direction control command.

[0025] A further technical solution involves generating translation control commands for the UAV based on the translation parameters required for UAV flight, including:

[0026] Based on the translation parameters required for UAV flight output by the horizontal flight prediction network, the output parameters are converted into UAV translation speed through linear scaling, and UAV translation control commands are generated.

[0027] A further technical solution involves inputting video frame images into an obstacle avoidance network, identifying obstacles in the video frame images, outputting the probability of the presence of obstacles in the images, and controlling the drone to perform obstacle avoidance operations when the probability is greater than a set threshold; at the same time, the obstacle avoidance network outputs the drone's rotational angular velocity to control the drone to achieve obstacle avoidance.

[0028] Secondly, this disclosure provides a visual navigation-based unmanned aerial vehicle (UAV) inspection system for substations, including a UAV and a ground station. The UAV includes an image acquisition module, an onboard processing module, and a first wireless communication module; the ground station includes a second wireless communication module, a data reading module, and a data processing and control module.

[0029] The image acquisition module is used to acquire video images of the substation site and transmit the acquired video images to the airborne processing module. The airborne processing module transmits the acquired video images to the ground station in the form of video stream data through the first wireless communication module.

[0030] The second wireless communication module of the ground station is used to acquire video stream data transmitted by the UAV. The data reading module is used to acquire video stream data collected by the UAV, preprocess the video stream data, acquire multiple consecutive video frame images, and transmit the video frame images to the data processing and control module. The data processing and control module is used to execute the above-mentioned substation UAV inspection method based on vision navigation, issue control commands, control the UAV to fly, and perform inspection.

[0031] Thirdly, this disclosure also provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps of the method described in the first aspect.

[0032] Fourthly, this disclosure also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps of the method described in the first aspect.

[0033] The above one or more technical solutions have the following beneficial effects:

[0034] 1. This invention provides a visual navigation-based UAV inspection method and system for substations. It acquires real-time video images captured by the UAV, constructs a navigation neural network including a feature extraction network, a yaw angle prediction network, and a horizontal flight prediction network, and learns from the visual images of the navigation neural network in the substation. This enables the UAV to recognize the pre-set inspection path and surrounding environmental obstacles, conduct autonomous flight and automatic substation inspection, eliminate the dependence on GPS navigation signals, and solve the problem of UAVs being affected by the quality of GPS signals in substations during substation inspections.

[0035] 2. The system provided by this invention includes a wireless transmission module and a data reading module, which enable the reading and processing of drone status information and camera video stream data using APKs commonly supported by drone manufacturers. This solves the problem that existing drones are semi-open platforms and cannot directly read drone status information and gimbal camera video stream data. Attached Figure Description

[0036] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0037] Figure 1 This is an overall flowchart of the visual navigation-based unmanned aerial vehicle (UAV) inspection method for substations as described in Embodiment 1 of the present invention;

[0038] Figure 2 This is a schematic diagram of the navigation neural network structure in Embodiment 1 of the present invention;

[0039] Figure 3 This is a schematic diagram illustrating the determination of the candidate rotation direction for UAV flight in Embodiment 1 of the present invention;

[0040] Figure 4 This is a schematic diagram of the obstacle avoidance network in Embodiment 1 of the present invention;

[0041] Figure 5 This is a schematic diagram of the visual navigation-based unmanned aerial vehicle (UAV) inspection system for substations as described in Embodiment 2 of the present invention;

[0042] Figure 6 This is the display interface of the UAV status display module in Embodiment 2 of the present invention. Detailed Implementation

[0043] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0045] Example 1

[0046] This embodiment provides a visual navigation-based unmanned aerial vehicle (UAV) inspection method for substations, such as... Figure 1 As shown, it includes the following steps:

[0047] The system acquires video stream data collected by the drone, preprocesses the video stream data, and obtains multiple consecutive video frame images.

[0048] The video frame image is input into the feature extraction network to extract the substation road environment features. The substation road environment features are then input into the yaw angle prediction network and the horizontal flight prediction network, respectively, and the yaw angle parameters of the substation road in each direction and the translation parameters required for UAV flight are output.

[0049] Based on the deflection angle parameters of the substation roads in each direction, and using a Gaussian mixture model, the yaw angle of the UAV is determined, and then the rotation direction control command of the UAV is generated; based on the translation parameters required for the flight of the UAV, the translation control command of the UAV is generated.

[0050] The drone is controlled to fly and perform inspections based on the drone's rotation direction control command and the drone's translation control command.

[0051] In this embodiment, a drone is used for substation inspection. The drone collects video stream data through its onboard camera and transmits the video stream data to the ground station. The ground station obtains the video stream data collected by the drone and performs preprocessing on the video stream data. This preprocessing includes, but is not limited to, filtering out noise and interference, improving contrast, enhancing edges, and geometric correction, to obtain multiple consecutive video frame images.

[0052] Based on the acquired video frame images, the UAV achieves autonomous navigation in the substation. The characteristic information for substation navigation is the concrete road that commonly surrounds the substation. In the video frame images, the angle between the road and the bottom edge of the image is taken as the yaw angle in the UAV flight control, and the deviation between the center point of the road and the center of the ground edge of the image is taken as the horizontal flight offset in the UAV flight control. Based on the yaw angle and the horizontal flight offset, the UAV is controlled to achieve autonomous navigation and inspection.

[0053] In this embodiment, the navigation neural network used in the visual navigation-based UAV inspection of substations consists of three subnetworks, such as... Figure 2 As shown, these are the feature extraction network, yaw angle prediction network, and horizontal flight prediction network, respectively. The video frame image is input into the feature extraction network to extract the substation road environment features. These features are then input into the yaw angle prediction network and the horizontal flight prediction network, respectively, outputting the yaw angle parameters of the substation road in each direction and the translation parameters required for UAV flight.

[0054] For the feature extraction network, since the road and surrounding environment differ significantly in substations, and road features are quite distinct, the feature extraction network is set as a ResNet V2 residual block to extract substation road environment features from the input video frame images. Using a relatively simple feature extraction network ensures accurate extraction of substation road environment features while maintaining real-time performance, thereby enabling real-time control of the UAV.

[0055] For the yaw angle prediction network, due to the presence of three-way intersections and crossroads in the substation road distribution, it is necessary to predict multiple road directions. In this embodiment, the substation road environment features are input into the yaw angle prediction network to obtain the yaw angle parameters of the substation roads in each direction. Then, a Gaussian mixture model is used to fit the final road direction. The probability distribution of the Gaussian mixture model representing the probability distribution of the substation road direction is shown in the following formula:

[0056]

[0057] In the above formula, x∈(-1,1) represents the range of road deflection angles in the image. The normalized value; P(x) represents the probability that the direction of the deflection angle is the direction of the road; , respectively, are the linear scaling scale, mean, and variance of the i-th component of the Gaussian mixture distribution, i.e., the yaw angle parameters of each substation road output by the yaw angle prediction network; n represents the number of road directions of the substation road. In this embodiment, n is set to 3 so that three different road directions at the substation road intersection can be predicted.

[0058] Meanwhile, since the road direction is the most important factor in this embodiment, the key to the Gaussian mixture probability distribution is the successful prediction of the road direction. Therefore, the variance of the Gaussian function is not important. Thus, this embodiment fixes the variance σ in the Gaussian probability distribution to simplify and accelerate training. Therefore, the yaw angle prediction network consists of a ResNet residual block and four fully connected layers. For each input feature, the yaw angle prediction network outputs 2n Gaussian mixture model parameters, that is, it outputs the yaw angle parameters of the substation road in n directions. and

[0059] For the horizontal flight prediction network, this network contains a ResNet residual block and three fully connected layers to predict the translation parameters required for UAV flight. This refers to the translation required to keep the drone above the road in images captured by the drone's onboard camera.

[0060] During the training of the aforementioned neural network, two loss functions are constructed. The loss function for the yaw angle prediction network output is the standard negative log-likelihood function, as shown in the following equation:

[0061]

[0062] In the above formula, D represents the direction label of the training dataset, j represents the j-th sample, j∈[1[N], and N is the total number of samples in the training dataset. This sample is a video frame image with direction labels. This is the output of the yaw angle prediction network.

[0063] The output of the horizontal flight prediction network is a fixed number, namely the translation parameters required for the drone's flight. Therefore, it is trained using only the mean squared error loss function, as shown in the following equation:

[0064]

[0065] In the above formula, T represents the translation label. To predict the translation label, i represents the i-th sample, i∈[1[N], and N is the total number of samples in the training dataset.

[0066] Then, based on the deflection angle parameters of the substation roads in each direction, the yaw angle of the UAV is determined using a Gaussian mixture model, and the rotation direction control command of the UAV is generated; based on the translation parameters required for the flight of the UAV, the translation control command of the UAV is generated.

[0067] After obtaining the predicted output of the UAV control neural network, it is necessary to calculate the final flight control output of the UAV, namely the rotation direction control command and the horizontal flight translation control command. In each control cycle, the predicted output of the yaw angle prediction network is read. Based on the Gaussian mixture model, the probability distribution of the road direction of each substation is obtained. The midpoint of the probability distribution segment corresponding to the part with a probability greater than a predetermined threshold is taken as the candidate rotation direction of the UAV. The UAV flight candidate rotation direction with the highest probability is selected as the output rotation direction, that is, the UAV yaw angle is determined. The rotation direction is linearly converted into the UAV rotation angular velocity through linear scaling, thus generating the UAV rotation direction control command.

[0068] Specifically, the yaw angle prediction neural network outputs the yaw angle parameters of the substation road in each direction. A Gaussian mixture model is then used to fit the final road direction, outputting the yaw angle of the road in the image captured by the UAV gimbal. The yaw angle range in the image is... in, 0 indicates the left side of the road, and 0 indicates the front side of the road. Representing the right side of the road, after normalizing this angle range to (-1, 1), -1, 0, and 1 represent the left, front, and right sides of the road, respectively. The yaw angle prediction neural network outputs multiple Gaussian probability functions. Adding these multiple Gaussian distributions yields a mixture of Gaussian distribution functions, such as... Figure 3 As shown, in this function, x∈(-1,1) is the normalized road deviation angle, and y is the probability of the road deviation angle, i.e. Figure 3 The vertical axis represents probability P, and the point (x[y]) represents the probability P when the road deviation angle is x. When the probability threshold is 1, the range of the x-axis corresponding to the curve with a probability greater than this threshold is -0.25 to 0.25. This segment is called the probability distribution segment, and the value corresponding to the midpoint of this probability distribution segment is taken as ( Figure 2 The value shown is 0, which represents the candidate rotation direction for UAV flight.

[0069] The yaw angle prediction neural network outputs the yaw angle parameters of the substation roads in each direction, obtaining the Gaussian mixture distribution function of each substation road in the image. This determines the candidate rotation direction for the UAV corresponding to each road, selecting the candidate rotation direction with the highest probability as the output rotation direction. Through linear scaling, the rotation direction is linearly converted into the UAV's rotation angular velocity, thus generating the UAV rotation direction control command. This command controls the UAV's rotation angular velocity until the UAV rotates to the direction the gimbal faces towards the road's extension, with the fuselage parallel to the road's centerline. At this point, the yaw angle prediction neural network's predicted output is 0, and after linear scaling, the UAV's rotation angular velocity is also 0. The UAV then stops rotating, achieving the desired yaw angle.

[0070] As another implementation method, if the UAV receives the substation road direction given by the user, the UAV selects the rotation direction that is closest to the substation road direction given by the user from the candidate rotation directions of the UAV flight, and outputs the rotation direction.

[0071] Simultaneously, in each control cycle, the output of the horizontal flight prediction network is read, and the predicted result is converted into the drone's translational speed, i.e., the drone's translational control command, through linear scaling. Similarly, the drone's translational speed is controlled by this command. When the drone flies directly above the road, the output of the horizontal flight prediction network becomes 0, and after linear scaling, the drone's translational speed also becomes 0. The drone then stops translating and reaches the desired position.

[0072] After receiving the rotation direction control command and the translation control command of the UAV, the control command can be sent to the UAV through the ground station, thereby controlling the UAV to fly and carry out inspection.

[0073] As another implementation method, since drone flight inherently carries certain risks, such as encountering pedestrians, trees, power lines, etc., which could cause property damage and damage to the drone, the ability to successfully navigate while avoiding obstacles is crucial for drone inspections. Therefore, this embodiment introduces a DroNet neural network as an obstacle avoidance network to achieve automatic obstacle avoidance for the drone. The structure of this obstacle avoidance network is as follows... Figure 4 As shown, the video frame image is input into the obstacle avoidance network. The obstacle avoidance network identifies obstacles in the image and outputs the probability P of the existence of obstacles in the image, that is, the probability that there are obstacles in front of the drone. When the probability is greater than a set threshold, the obstacle avoidance operation is performed. The obstacle avoidance network outputs the drone's rotational angular velocity V to control the drone to achieve obstacle avoidance.

[0074] When GPS information is poor, the visual navigation-based UAV inspection method described in this embodiment can automatically inspect substations based on video stream data collected by the UAV, eliminating the dependence on GPS navigation signals and solving the problem of UAVs being affected by the quality of GPS signals at the substation during substation inspections, thus ensuring the safe operation of "unmanned" substations.

[0075] Example 2

[0076] Considering that the drones currently in use are not open platforms, although the drone's status information can be read through the official Android APP and the video stream data captured by the gimbal camera can be output and displayed as HDMI, other devices or platforms cannot directly process and predict the HDMI output data. Therefore, this embodiment presents a substation drone inspection system based on visual navigation, which reads drone data through a built software and hardware platform for subsequent processing.

[0077] This embodiment provides a visual navigation-based substation drone inspection system, including a drone and a ground station, such as... Figure 5 As shown, the UAV includes an image acquisition module, an airborne processing module, and a first wireless communication module; the ground station includes a second wireless communication module, a data reading module, and a data processing and control module.

[0078] The image acquisition module is used to acquire video images of the substation site and transmit the acquired video images to the airborne processing module. The airborne processing module transmits the acquired video images to the ground station in the form of video stream data through the first wireless communication module.

[0079] The second wireless communication module of the ground station is used to acquire video stream data transmitted by the UAV. The data reading module is used to acquire video stream data collected by the UAV, preprocess the video stream data, acquire multiple consecutive video frame images, and transmit the video frame images to the data processing and control module. The data processing and control module is used to execute the substation UAV inspection method based on visual navigation as described in Embodiment 1, issue control commands, control the UAV to fly, and perform inspection.

[0080] In this embodiment, the drone is a drone suitable for providing APK development programs, such as the DJI M100 drone. The drone transmits data through the first wireless communication module, transmitting video stream data and drone status data captured by the camera to the ground station. The ground station receives the transmitted data information through the second wireless communication module, obtains the video stream data output by the drone through the data reading module (i.e., video capture card), and preprocesses the drone video stream data into video frame images for subsequent data processing.

[0081] In another implementation, the ground station includes mobile control software and ground station control software. The UAV transmits video stream data captured by its camera to the mobile control software via a first communication unit. The mobile control software, developed using DJI Mobile APK, transmits UAV status data to the ground station control software via TCP socket. Simultaneously, it uses an image acquisition card to transmit video stream data captured by the gimbal camera. The ground control software, implemented using QT C++, communicates with the mobile control software to obtain UAV status data, acquires the video stream data output by the UAV via the video acquisition card, and preprocesses the UAV video stream data into image frames for subsequent prediction.

[0082] Specifically, in this embodiment, the UAV's onboard image acquisition module first acquires video images of the substation site. Then, the image acquisition module uploads the acquired video images to the onboard processing module. The onboard processing module communicates with the ground station's second wireless communication module through the first wireless communication module, transmitting the acquired video images to the ground station. At this time, the output is an HDMI video stream. Then, the data reading module (i.e., the video capture card) converts the HDMI video stream into USB camera video stream data. Thus, the ground station can obtain the video frame images captured by the UAV by reading the USB camera video stream data. The video frame images are then sent to the data processing and control module for prediction calculation, executing the method described in Embodiment 1 above, to realize automatic inspection of the UAV substation.

[0083] In the data reading module, the ground station reads the USB video stream data output by the video capture card at a frequency of 20Hz using OpenCV. After decomposing the video stream into video frame images, the images are saved as local files on the ground station. File locks are then used to lock and unlock the local images, enabling multi-process read and write operations on the local image files, thus achieving the reading and writing of video streams captured by the UAV gimbal camera. Simultaneously, to read the UAV's flight speed, altitude, and other status information, a TCP server is set up on the ground station, establishing a TCP connection with the mobile control software to read the UAV's status data. Since the UAV's status data can be represented using only short strings, using TCP communication within a local area network can meet the real-time requirements.

[0084] The system described in this embodiment also includes a UAV motion control module, which is located in a ground station and is used to input manual control information. The data processing and control module of the ground station can read the keyboard input information, convert the keyboard input information into UAV flight control commands, and send them to the mobile control software. The mobile control software sends control commands to the UAV to realize manual control of the UAV's flight status.

[0085] The system described in this embodiment also includes a UAV substation inspection control module. The UAV substation inspection control module is set in the ground station and is used to input path information. When the GPS information is good, by setting path points on the map, the data processing and control module of the ground station calculates the path points as the flight path of the UAV through linear interpolation, calculates the UAV flight control command based on the path, and sends the control command to the mobile control software. The mobile control software sends the control command to the UAV, thereby enabling the UAV to fly along the path points for inspection.

[0086] When GPS information is poor, the UAV substation inspection mode can be switched to a vision-based navigation inspection method. The ground station's data processing and control module executes the vision-based navigation UAV substation inspection method described in Example 1 to perform the inspection. At the same time, navigation inspection parameters, such as obstacle recognition confidence threshold, can be modified in real time to correct the vision-based navigation UAV inspection effect at any time.

[0087] The system described in this embodiment also includes a drone status display module, in which, for example... Figure 6 As shown, it can display video stream images captured by the drone's gimbal camera, drone flight status information, and surrounding map information in real time, as well as the parameter display and settings in each of the above modules, realizing simple and quick human-computer interaction.

[0088] This embodiment solves the problem of not being able to directly read drone status information and gimbal camera video stream data by using APK (Android application package) commonly supported by drone manufacturers to read and process drone status information and camera video stream data. At the same time, through visual image learning of substations, the drone can recognize the pre-set inspection path and surrounding environmental obstacles, thereby getting rid of dependence on GPS navigation signals and solving the problem of drones being affected by the quality of substation GPS signals during substation inspections.

[0089] Example 3

[0090] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the steps in the vision-based navigation-based UAV inspection method for substations as described above.

[0091] Example 4

[0092] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps in the vision-navigation-based UAV inspection method for substations as described above.

[0093] The steps and methods involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0094] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0096] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for unmanned aerial vehicle (UAV) inspection of substations based on vision navigation, characterized in that, include: The system acquires video stream data collected by the drone, preprocesses the video stream data, and obtains multiple consecutive video frame images. The video frame image is input into the feature extraction network to extract the substation road environment features. The substation road environment features are then input into the yaw angle prediction network and the horizontal flight prediction network, respectively, and the yaw angle parameters of the substation road in each direction and the translation parameters required for UAV flight are output. Based on the deflection angle parameters of the substation roads in each direction, and using a Gaussian mixture model, the yaw angle of the UAV is determined, and then the rotation direction control command of the UAV is generated; based on the translation parameters required for the flight of the UAV, the translation control command of the UAV is generated. Based on the drone's rotation direction control command and drone translation control command, the drone is controlled to fly and perform inspections; The method involves determining the yaw angle of the UAV based on the deflection angle parameters of the substation road in each direction, using a Gaussian mixture model, and then generating UAV rotation direction control commands, including: Based on the yaw angle parameters of the substation roads in each direction output by the yaw angle prediction network, the probability distribution of each substation road direction is obtained based on the Gaussian mixture model. The midpoint of the probability distribution segment corresponding to the part with a probability greater than a predetermined threshold is taken as the candidate rotation direction for UAV flight. The highest probability candidate rotation direction of the UAV flight is selected as the output rotation direction, and the output rotation direction is used as the UAV yaw angle; The rotation direction is linearly converted into the UAV's rotation angular velocity by linear scaling, thereby generating the UAV's rotation direction control command.

2. The method for substation unmanned aerial vehicle (UAV) inspection based on vision navigation as described in claim 1, characterized in that, The yaw angle prediction network consists of a ResNet residual block and four fully connected layers. For each input feature, the yaw angle prediction network outputs... n Deflection parameters of substation roads in each direction .

3. The method for unmanned aerial vehicle (UAV) inspection of substations based on vision navigation as described in claim 2, characterized in that, The loss function of the yaw angle prediction network is the standard negative log-likelihood function, and the formula is: In the above formula, Directional labels for the training dataset. j Indicates the first j One sample, j ∈[1, N ], The total number of samples in the training dataset. This indicates the range of road deflection angles in the image. The normalized value, This represents the probability that the direction of the deflection angle is the direction of the road. The output of the yaw angle prediction network is the yaw angle parameters of each substation road output by the yaw angle prediction network. n This indicates the number of road directions for the substation.

4. The method for unmanned aerial vehicle (UAV) inspection of substations based on vision navigation as described in claim 1, characterized in that, The horizontal flight prediction network comprises a ResNet residual block and three fully connected layers, used to predict the translation parameters required for UAV flight. .

5. The method for substation unmanned aerial vehicle (UAV) inspection based on vision navigation as described in claim 4, characterized in that, The loss function of the horizontal flight prediction network is the mean squared error loss, and the formula is: In the above formula, For translation labels, To predict translation labels, i Indicates the first i One sample, i ∈[1, N ], This represents the total number of samples in the training dataset.

6. The method for unmanned aerial vehicle (UAV) inspection of substations based on vision navigation as described in claim 1, characterized in that, The process of generating UAV translation control commands based on the translation parameters required for UAV flight includes: Based on the translation parameters required for UAV flight output by the horizontal flight prediction network, the output parameters are converted into UAV translation speed through linear scaling, and UAV translation control commands are generated.

7. A visual navigation-based unmanned aerial vehicle (UAV) inspection system for substations, characterized in that, The system includes a drone and a ground station. The drone includes an image acquisition module, an onboard processing module, and a first wireless communication module. The ground station includes a second wireless communication module, a data reading module, and a data processing and control module. The image acquisition module is used to acquire video images of the substation site and transmit the acquired video images to the airborne processing module. The airborne processing module transmits the acquired video images to the ground station in the form of video stream data through the first wireless communication module. The second wireless communication module of the ground station is used to acquire video stream data transmitted by the UAV. The data reading module is used to acquire video stream data collected by the UAV, preprocess the video stream data, acquire multiple consecutive video frame images, and transmit the video frame images to the data processing and control module. The data processing and control module is used to execute the substation UAV inspection method based on visual navigation as described in any one of claims 1-6, issue control commands, control the UAV to fly, and perform inspection.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, complete the steps of a visual navigation-based unmanned substation inspection method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of a visual navigation-based unmanned aerial vehicle (UAV) inspection method for substations as described in any one of claims 1-6.

Images