Visual perception driven distribution network unmanned aerial vehicle inspection method and device, equipment and medium

Abstract

According to the visual perception driven distribution network unmanned aerial vehicle inspection method and device, the equipment and the medium, intelligent flight control and a path optimization algorithm are combined, autonomous flight and dynamic path planning of the unmanned aerial vehicle can be achieved, and compared with a traditional method depending on manual operation and route presetting, the visual perception driven distribution network unmanned aerial vehicle inspection method has the advantages that the efficiency is high; the method greatly improves the autonomy and efficiency of routing inspection, remarkably reduces the workload of manual intervention and early-stage preparation, has the capability of real-time sensing and quick response to environment change, can automatically adjust the path and optimize the task through the analysis and processing of real-time data, effectively deals with the complex power grid environment, and improves the efficiency of routing inspection. And the real-time performance of the inspection task and the decision intelligent level are obviously improved. The target component of the power distribution network can be accurately identified by using the distribution network target detection model, the target component is automatically aligned, the focal length is adjusted for shooting, the reliability and accuracy of shooting in the inspection task are remarkably improved, and intelligent alignment and high-precision shooting are realized.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, in particular to the field of visual processing technology, and discloses a distribution network drone inspection method and device, equipment, and medium driven by visual perception. Background Art

[0002] In recent years, with the development of drone technology and its deepening application in the power industry, drone power distribution network inspection has become an important means to improve the efficiency of power grid operation and maintenance and ensure the safety of power supply. At present, drone inspection systems can be equipped with advanced equipment such as high-definition cameras, infrared thermal imagers, and lidar to achieve all-round and no-dead-angle monitoring of transmission lines and related facilities. The application of these technologies has greatly improved the efficiency and accuracy of inspection work and reduced the risks brought by manual inspection. Although drone power inspection has made significant progress, there are still some challenges and limitations in practical applications:

[0003] High cost: UAVs equipped with high-precision sensors such as LiDAR can provide detailed three-dimensional terrain and structural information, greatly improving the quality of inspection data. However, these advanced sensors are expensive, resulting in a significant increase in the procurement cost of the entire UAV system. For small and medium-sized power companies with limited resources, this has become a major financial burden, limiting their adoption of more advanced inspection technologies.

[0004] Complex manual operation: Although drones have simplified many inspection processes, they still require professional operation when performing certain tasks that require highly precise control. For example, adjusting the drone's posture to obtain the best viewing angle in a complex environment or responding immediately to emergencies all place high demands on the operator's skill level. This also means that training qualified operators becomes a continuous investment.

[0005] Insufficient flexibility: In order to ensure that the drone flies safely along the predetermined path and completes the inspection task, detailed planning is usually required before the flight, that is, route planning relies on preliminary preparation. Although this method can ensure the safety and stability of the flight, once a temporary change (such as a new obstacle) is encountered, the route needs to be readjusted, which reduces the flexibility and response speed of the overall operation. Summary of the invention

[0006] The present disclosure at least provides a visual perception-driven distribution network drone inspection method and device, equipment, and medium to reduce power inspection costs, improve the efficiency and intelligence of distribution network inspections, simplify manual operation processes, and improve inspection flexibility and response speed.

[0007] According to one aspect of the present disclosure, a visual perception driven distribution network drone inspection method is provided, comprising:

[0008] S110, obtaining a video image captured by a gimbal camera mounted on the drone, drone position information collected by an RTK module mounted on the drone, and drone attitude information; wherein the drone attitude information includes a gimbal angle of the drone;

[0009] S120, using the trained distribution network target detection model to process the video image, and identify the target tower in the current video frame;

[0010] S130, determining the second gimbal angle of the drone at the next moment according to the position information of the target pole tower in the current video frame and the first gimbal angle of the drone at the current moment; determining the flight distance of the drone from the current moment to the next moment according to the position information of the drone at the current moment and the flight speed of the drone; and determining the gimbal attitude change data according to the first gimbal angle and the second gimbal angle; determining the horizontal distance of the drone to the target tower according to the gimbal attitude change data, the flight distance, and the first gimbal angle; determining the position information of the target tower according to the horizontal distance and the drone position information at the current moment; determining the flight trajectory information of the drone, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment according to the position information of the target tower and the drone position information at the current moment, so as to control the drone to fly according to the flight trajectory information, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment;

[0011] S140, when the drone reaches directly above the target tower, determine the position of a first photographing point according to the drone position information, the heading angle of the drone, and the height of the target tower, and use the position of the first photographing point to control the drone to fly to the first photographing point;

[0012] S150, when the UAV arrives at the target photographing point, obtain the current view captured by the gimbal camera, process the current view using the distribution network target detection model, and determine the minimum bounding box of each target component according to the processing result; use the minimum bounding box of each target component to respectively determine the position and size of each target component in the current view; for each target component, based on the position of the target component in the current view and the current focal length of the gimbal camera, determine the angular offset of the target component relative to the gimbal of the UAV, based on the current gimbal angle of the UAV and the angular offset of the target component relative to the gimbal of the UAV, determine the gimbal angle of the UAV to photograph the target component, based on the current zoom factor of the gimbal camera and the size of the target component in the current view, determine the zoom factor of the UAV to photograph the target component, and control the gimbal camera to photograph the target component according to the gimbal angle and zoom factor of the UAV to photograph the target component; wherein the target photographing point includes one of the four photographing points;

[0013] S160, when the drone has captured images of all target components at the current target photographing point, determining the position of the next photographing point according to the drone position information, the heading angle of the drone, and the height of the target tower, and controlling the drone to fly to the next photographing point using the position of the next photographing point;

[0014] S170, looping through at least one of steps S150 and S160 until the drone has captured images of the target components corresponding to the four photographing points; wherein the four photographing points include a front photographing point, a right photographing point, a rear photographing point, and a left photographing point in sequence;

[0015] S180, control the UAV to fly to the target safe height and complete the task of photographing the target tower.

[0016] In a possible implementation manner, determining the horizontal distance from the drone to the target tower pole according to the gimbal attitude change data, the flight distance, and the first gimbal angle includes:

[0017] The horizontal distance is determined using the following formula:

[0018]

[0019] Wherein, α represents the first gimbal angle, β represents the second gimbal angle, tanβ-tanα represents the gimbal attitude change data, v*dt represents the flight distance, and S represents the horizontal distance.

[0020] In a possible implementation, determining the position of the first photo taking point according to the drone position information, the flight direction, and the height of the target tower includes:

[0021] Use the following formula to determine the location of the first photo point:

[0022]

[0023] Among them, (lat 1 ,lon 1 ) represents the position information of the UAV, d represents the height of the target tower, heading represents the heading angle (angle) of the current UAV, and R represents the radius of the earth ellipsoid.

[0024] In a possible implementation, the determining the position and size of each target component in the current view using the minimum bounding box of each target component includes:

[0025] For each target component p, the position of the corresponding target component in the current view is determined using the following formula:

[0026] p xy =min(obb)

[0027] Among them, p xy (p x ,p y ) represents the position of the target component in the current view, and obb represents the coordinates of the four vertices of the minimum bounding box of the target component;

[0028] For each target component, the size of the corresponding target component in the current view is determined using the following formula:

[0029] obb h =dis(obb)

[0030] Among them, obb h Indicates the size of the target component in the current view.

[0031] In a possible implementation, determining the angular offset of the target component relative to the gimbal of the drone based on the position of the target component in the current view and the current focal length of the gimbal camera includes:

[0032]

[0033] delta_α=arctan(d y / fl_px)

[0034] Among them, P yIndicates the position of the target component in the y-axis direction in the current view, H indicates the height of the current view, d y is the offset from the target component to the center of the image, fl_px represents the current focal length of the gimbal camera, and delta_α represents the angular offset of the target component relative to the gimbal of the drone.

[0035] In a possible implementation, determining the gimbal angle of the drone for photographing the target component based on the current gimbal angle of the drone and the angular offset of the target component relative to the gimbal of the drone includes:

[0036] next_β=delta_α+cur_β

[0037] Among them, cur_β represents the current gimbal angle of the drone, delta_α represents the angular offset of the target component relative to the drone's gimbal, and next_β represents the gimbal angle of the drone when shooting the target component.

[0038] In a possible implementation, determining the zoom factor of the target component photographed by the drone based on the current zoom factor of the gimbal camera and the size of the target component in the current view includes:

[0039]

[0040] Among them, nextR represents the zoom factor of the drone shooting the target component, and curR represents the current zoom factor of the gimbal camera.

[0041] According to another aspect of the present disclosure, a visual perception driven distribution network drone inspection device is provided, comprising:

[0042] A data acquisition module, used to acquire video images captured by a pan-tilt camera mounted on a drone, drone position information collected by an RTK module mounted on the drone, and drone attitude information; wherein the drone attitude information includes the pan-tilt angle of the drone;

[0043] A model processing module is used to process the video image using the trained distribution network target detection model to identify the target tower in the current video frame;

[0044] A path planning module, for determining the second gimbal angle of the drone at the next moment according to the position information of the target pole tower in the current video frame and the first gimbal angle of the drone at the current moment; determining the flight distance of the drone from the current moment to the next moment according to the position information of the drone at the current moment and the flight speed of the drone; and determining the gimbal attitude change data according to the first gimbal angle and the second gimbal angle; determining the horizontal distance of the drone to the target tower according to the gimbal attitude change data, the flight distance, and the first gimbal angle; determining the position information of the target tower according to the horizontal distance and the drone position information at the current moment; determining the flight trajectory information of the drone, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment according to the position information of the target tower and the drone position information at the current moment, so as to control the drone to fly according to the flight trajectory information, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment;

[0045] A photographing point determination module is used to determine the position of a first photographing point according to the position information of the drone, the heading angle of the drone, and the height of the target tower when the drone reaches the top of the target tower, so as to control the drone to fly to the first photographing point by using the position of the first photographing point;

[0046] A precision photographing module, used for obtaining the current view captured by the gimbal camera when the UAV reaches the target photographing point, processing the current view using the distribution network target detection model, and determining the minimum bounding box of each target component according to the processing result; using the minimum bounding box of each target component to respectively determine the position and size of each target component in the current view; for each target component, based on the position of the target component in the current view and the current focal length of the gimbal camera, determining the angular offset of the target component relative to the gimbal of the UAV, based on the current gimbal angle of the UAV and the angular offset of the target component relative to the gimbal of the UAV, determining the gimbal angle of the UAV for photographing the target component, based on the current zoom factor of the gimbal camera and the size of the target component in the current view, and controlling the gimbal camera to photograph the target component according to the gimbal angle and zoom factor of the UAV for photographing the target component; wherein the target photographing point includes one of the four photographing points;

[0047] The photographing point determination module is further used to determine the position of the next photographing point according to the position information of the drone, the heading angle of the drone, and the height of the target tower when the drone has taken images of all target components at the current target photographing point, so as to control the drone to fly to the next photographing point by using the position of the next photographing point;

[0048] The precision photographing module is further used to photograph the target component corresponding to each photographing point until the drone has finished photographing the target components corresponding to the four photographing points; wherein the four photographing points include the front photographing point, the right photographing point, the rear photographing point, and the left photographing point in sequence;

[0049] The homing module is used to control the drone to fly to the target safe altitude and complete the task of photographing the target tower.

[0050] According to another aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0051] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned methods is implemented.

[0052] The visual perception driven distribution network drone inspection method, device, equipment, and medium disclosed in the present invention can identify the target tower, determine the horizontal distance from the drone to the target tower, and use the horizontal distance and the drone position information to determine the position information of the target tower; use the position information of the target tower and the drone position information at the current moment to perform path planning, determine the flight trajectory information of the drone, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment, so as to control the drone to fly according to the flight trajectory information, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment, until the drone flies directly above the target tower. The present invention combines intelligent flight control and path optimization algorithms to realize autonomous flight and dynamic path planning of drones. Compared with the traditional method that relies on manual operation and preset routes, it greatly improves the autonomy and efficiency of inspection, significantly reduces the workload of manual intervention and preliminary preparation, and has the ability to perceive and respond quickly to environmental changes in real time. Through the analysis and processing of real-time data, drones can autonomously adjust paths and optimize tasks, effectively cope with complex power grid environments, and significantly improve the real-time nature of inspection tasks and the level of intelligent decision-making. The present invention uses the distribution network target detection model to accurately identify the distribution network environment and its accessories, determine the target components, automatically align the target components and adjust the focal length for shooting. This function significantly improves the reliability and accuracy of shooting in inspection tasks, especially in the multi-angle and multi-factor detection of targets, effectively meeting the high requirements of complex power grid equipment inspection, and realizing intelligent alignment and high-precision shooting. In short, the technical solution of the present invention realizes accurate detection, path optimization and intelligent inspection of key components of the distribution network.

[0053] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0055] Figure 1 is a flow chart of the distribution network drone inspection method driven by visual perception according to the present disclosure;

[0056] Figure 2 is a schematic diagram of calculating the horizontal distance according to an embodiment of the present disclosure;

[0057] Figure 3 is an information schematic diagram of a PTZ according to an embodiment of the present disclosure;

[0058] Figure 4 It is a structural schematic diagram of a distribution network drone inspection device driven by visual perception according to the present disclosure;

[0059] Figure 5 is a schematic structural diagram of an electronic device according to the present disclosure. DETAILED DESCRIPTION

[0060] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0061] The present invention provides a visual perception driven distribution network UAV inspection method, device, equipment and medium in view of the defects of high cost, low efficiency and intelligence of distribution network inspection, complex manual operation process, poor inspection flexibility and response speed of current UAV power inspection. The present invention combines intelligent flight control and path optimization algorithm to realize autonomous flight and dynamic path planning of UAV. Compared with the traditional method relying on manual operation and preset routes, it greatly improves the autonomy and efficiency of inspection, significantly reduces the workload of manual intervention and preliminary preparation, and has the ability to perceive and respond to environmental changes in real time. Through the analysis and processing of real-time data, the UAV can autonomously adjust the path and optimize the task, effectively cope with the complex power grid environment, and significantly improve the real-time and intelligent decision-making level of the inspection task. The present invention uses the distribution network target detection model to accurately identify the distribution network environment and its accessories, determine the target component, automatically align the target component and adjust the focal length for shooting. This function significantly improves the reliability and accuracy of shooting in the inspection task, especially in the multi-angle and multi-factor detection of the target object, effectively meets the high requirements of complex power grid equipment inspection, and realizes intelligent alignment and high-precision shooting.

[0062] The technical solution of the present disclosure is described below through specific embodiments.

[0063] like Figure 1 As shown, it is a flow chart of the visual perception driven distribution network drone inspection method of this embodiment. The execution subject of this embodiment is a computing device or component with data processing capability. The specific method of this embodiment may include the following steps:

[0064] S110, obtaining video images captured by a gimbal camera mounted on the drone, drone position information collected by an RTK module mounted on the drone, and drone attitude information; wherein the drone attitude information includes the gimbal angle of the drone.

[0065] This step monitors the data collected by the gimbal camera and RTK module mounted on the drone in real time.

[0066] S120: Process the video image using the trained distribution network target detection model to identify the target tower in the current video frame.

[0067] Before executing this step, the video image needs to be preprocessed, and then the preprocessed video image is processed using the distribution network target detection model.

[0068] The distribution network target detection model not only identifies the target tower in the video image, but also can self-detect key components such as conductors, insulators, and wire clamps on the tower (i.e., subsequent target components).

[0069] Steps S110 and S120 implement the detection of key targets (i.e., target towers) and component points (i.e., subsequent target components) using a self-developed deep learning model (i.e., distribution network target detection model) after receiving the real-time video stream.

[0070] S130, planning the flight trajectory of the drone, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment according to the gimbal angle of the drone at the current moment and the position of the target tower: determining the second gimbal angle of the drone at the next moment according to the position information of the target tower in the current video frame and the first gimbal angle of the drone at the current moment; determining the flight distance of the drone from the current moment to the next moment according to the position information of the drone at the current moment and the flight speed of the drone; and determining the gimbal attitude change data according to the first gimbal angle and the second gimbal angle; determining the horizontal distance of the drone to the target tower according to the gimbal attitude change data, the flight distance, and the first gimbal angle; determining the position information of the target tower according to the horizontal distance and the drone position information at the current moment; determining the flight trajectory information of the drone, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment according to the position information of the target tower and the drone position information at the current moment, so as to control the drone to fly according to the flight trajectory information, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment.

[0071] Specifically, Figure 2 As shown, the horizontal distance S can be determined using the following formula:

[0072]

[0073] Wherein, α represents the first gimbal angle, β represents the second gimbal angle, tanβ-tanα represents the gimbal attitude change data, v*dt represents the flight distance, and S represents the horizontal distance.

[0074] This step can also obtain the approximate position of the target tower based on the estimated horizontal distance S and the current drone POS (i.e., drone position information), drone attitude and other information, and plan the subsequent flight trajectory of the drone, giving the drone's flight direction, attitude and other information at the next moment.

[0075] This step realizes tracking of the tower target and planning of the flight trajectory to the target tower based on the detection results (i.e. the position information of the target tower in the current video frame) and the real-time attitude information of the drone (i.e. the first gimbal angle mentioned above).

[0076] S140. When the drone reaches directly above the target tower, determine the position of the first photo point based on the drone position information, the heading angle of the drone, and the height of the target tower, and use the position of the first photo point to control the drone to fly to the first photo point.

[0077] The first photo taking point here can be the point in front of the target tower.

[0078] Use the following formula to determine the location of the first photo point:

[0079]

[0080] Among them, (lat 1 ,lon 1 ) represents the position information of the drone, that is, the latitude and longitude of the drone's hovering position directly above the target tower, d represents the height of the target tower, heading represents the current heading angle (angle) of the drone, and R represents the radius of the earth's ellipsoid. (lat 2 ,lon 2 ) represents the longitude and latitude of the first photo-taking point.

[0081] The value of d can be 2 meters, which represents the horizontal distance between the next photo point and the current position.

[0082] S150, aligning and photographing the target component at the photographing point: when the UAV arrives at the target photographing point, obtaining the current view photographed by the gimbal camera, processing the current view by using the distribution network target detection model, and determining the minimum bounding box of each target component according to the processing result; using the minimum bounding box of each target component to respectively determine the position and size of each target component in the current view; for each target component, based on the position of the target component in the current view and the current focal length of the gimbal camera, determining the angular offset of the target component relative to the gimbal of the UAV, based on the current gimbal angle of the UAV and the angular offset of the target component relative to the gimbal of the UAV, determining the gimbal angle of the UAV for photographing the target component, based on the current zoom factor of the gimbal camera and the size of the target component in the current view, determining the zoom factor of the UAV for photographing the target component, and controlling the gimbal camera to photograph the target component according to the gimbal angle and zoom factor of the UAV for photographing the target component; wherein the target photographing point includes one of the four photographing points.

[0083] like Figure 3 As shown, for each target component p, the position of the corresponding target component in the current view is determined using the following formula:

[0084] p xy=min(obbh)

[0085] Among them, p xy Indicates the position of the target component in the current view, and obb indicates the coordinates of the four vertices of the minimum bounding box of the target component. xy Including p x and p y , that is, the x-axis and y-axis coordinates of the target component in the current view.

[0086] For each target component, the size of the corresponding target component in the current view is determined using the following formula:

[0087] obb h = = dis(obb)

[0088] Among them, obb h Indicates the size of the target component in the current view, which can be the long side of the minimum bounding box.

[0089] Based on the position of the target component in the current view and the current focal length of the gimbal camera, the angular offset of the target component relative to the gimbal of the drone is determined, including:

[0090]

[0091] delta_α=arctan(d y / fl_px)

[0092] Among them, P y Indicates the position of the target component in the y-axis direction in the current view, H indicates the height of the current view, fl_px indicates the current focal length of the gimbal camera, delta_α indicates the angular offset of the target component relative to the gimbal of the drone, d y is the offset from the target part to the center of the image.

[0093] Based on the current gimbal angle of the drone and the angular offset of the target component relative to the gimbal of the drone, the gimbal angle of the drone for photographing the target component is determined, including:

[0094] next_β=delta_α+cur_β

[0095] Among them, cur_β represents the current gimbal angle of the drone, delta_α represents the angular offset of the target component relative to the drone's gimbal, and next_β represents the gimbal angle of the drone when shooting the target component.

[0096] Based on the current zoom factor of the gimbal camera and the size of the target component in the current view, the zoom factor of the target component photographed by the drone is determined, including:

[0097]

[0098] Among them, nextR represents the zoom factor of the target component photographed by the drone, curR represents the current zoom factor of the gimbal camera, and 0.4 is the set threshold.

[0099] After the drone arrives at the photo-taking point, it calculates the position p of each target component in the current view based on the detected target components. xy and size obb h , based on the current drone's attitude information (azimuth angle yaw, gimbal pitch angle cur_β), calculate the drone's attitude corresponding to each target component (i.e., the angle of the drone's gimbal), and calculate the gimbal angle (next_β) and camera zoom factor (nextR), as shown in the above formula, and then control the drone to take fine photos of the target components in the current view in a certain order.

[0100] S160. When the drone has captured images of all target components at the current target photographing point, the position of the next photographing point is determined according to the drone position information, the heading angle of the drone, and the height of the target tower, so as to control the drone to fly to the next photographing point using the position of the next photographing point.

[0101] S170, looping through at least one of steps S150 and S160 until the drone has captured images of the target components corresponding to the four photographing points; wherein the four photographing points include a front photographing point, a right photographing point, a rear photographing point, and a left photographing point in sequence.

[0102] Steps S140-S170 realize that after the UAV reaches the top of the target tower, it calculates the photographing points on the four directions of the target tower based on the current POS (i.e., the UAV position information), flight direction, and the height of the target tower, and plans the UAV flight sequence and trajectory in the clockwise order of front-right-back-left of the current UAV; then the target components are accurately aligned and photographed at each point through the alignment algorithm.

[0103] S180, control the UAV to fly to the target safe height and complete the task of photographing the target tower.

[0104] After completing the shooting task of the target tower, you can continue with the inspection task of subsequent towers.

[0105] Based on the same inventive concept, the present disclosure provides a visual perception driven distribution network drone inspection device, the steps performed by the components of the device are the same or similar to the above method, so similar parts are not repeated. Figure 4As shown, the visual perception driven distribution network drone inspection device of this embodiment includes:

[0106] The data acquisition module 410 is used to acquire the video images taken by the gimbal camera mounted on the drone, the drone position information collected by the RTK module mounted on the drone, and the drone attitude information; wherein the drone attitude information includes the gimbal angle of the drone.

[0107] The model processing module 420 is used to process the video image using the trained distribution network target detection model to identify the target tower in the current video frame.

[0108] The path planning module 430 is used to determine the second gimbal angle of the drone at the next moment according to the position information of the target tower in the current video frame and the first gimbal angle of the drone at the current moment; determine the flight distance of the drone from the current moment to the next moment according to the position information of the drone at the current moment and the flight speed of the drone; and determine the gimbal attitude change data according to the first gimbal angle and the second gimbal angle; determine the horizontal distance of the drone to the target tower according to the gimbal attitude change data, the flight distance, and the first gimbal angle; determine the position information of the target tower according to the horizontal distance and the drone position information at the current moment; determine the flight trajectory information of the drone, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment according to the position information of the target tower and the drone position information at the current moment, so as to control the drone to fly according to the flight trajectory information, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment.

[0109] The photographing point determination module 440 is used to determine the position of the first photographing point according to the position information of the drone, the heading angle of the drone, and the height of the target tower when the drone reaches the top of the target tower, so as to control the drone to fly to the first photographing point by using the position of the first photographing point;

[0110] The precision photographing module 450 is used to obtain the current view captured by the gimbal camera when the UAV arrives at the target photographing point, process the current view using the distribution network target detection model, and determine the minimum bounding box of each target component according to the processing result; use the minimum bounding box of each target component to respectively determine the position and size of each target component in the current view; for each target component, based on the position of the target component in the current view and the current focal length of the gimbal camera, determine the angular offset of the target component relative to the gimbal of the UAV, based on the current gimbal angle of the UAV and the angular offset of the target component relative to the gimbal of the UAV, determine the gimbal angle of the UAV to photograph the target component, based on the current zoom factor of the gimbal camera and the size of the target component in the current view, determine the zoom factor of the UAV to photograph the target component, and control the gimbal camera to photograph the target component according to the gimbal angle and zoom factor of the UAV to photograph the target component; wherein the target photographing point includes one of the four photographing points.

[0111] The photographing point determination module 440 is also used to determine the position of the next photographing point according to the drone position information, the drone heading angle, and the height of the target tower when the drone has captured images of all target components at the current target photographing point, so as to use the position of the next photographing point to control the drone to fly to the next photographing point.

[0112] The precision photographing module 450 is also used to photograph the target components corresponding to each photographing point until the drone has taken images of the target components corresponding to the four photographing points; wherein the four photographing points include the front photographing point, the right photographing point, the rear photographing point, and the left photographing point in sequence.

[0113] The homing module 460 is used to control the UAV to fly to the target safe height to complete the task of photographing the target tower.

[0114] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a computer-readable storage medium.

[0115] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0116] like Figure 5 As shown, the device 500 includes a computing unit 510, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 520 or a computer program loaded from a storage unit 580 into a random access memory (RAM) 530. In the RAM 530, various programs and data required for the operation of the device 500 can also be stored. The computing unit 510, the ROM 520, and the RAM 530 are connected to each other via a bus 540. An input / output (I / O) interface 550 is also connected to the bus 540.

[0117] A number of components in the device 500 are connected to the I / O interface 550, including: an input unit 560, such as a keyboard, a mouse, etc.; an output unit 570, such as various types of displays, speakers, etc.; a storage unit 580, such as a disk, an optical disk, etc.; and a communication unit 590, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 590 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0118] The computing unit 510 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 510 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 510 performs the various methods and processes described above. For example, in some embodiments, any of the above methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 580. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM520 and / or communication unit 590. When the computer program is loaded into RAM530 and executed by the computing unit 510, one or more steps of any of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 510 may be configured to perform any of the methods described above in any other appropriate manner (e.g., by means of firmware).

[0119] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0121] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A visual perception driven distribution network drone inspection method, characterized in that: include: S110, obtaining a video image captured by a gimbal camera mounted on the drone, drone position information collected by an RTK module mounted on the drone, and drone attitude information; wherein the drone attitude information includes a gimbal angle of the drone; S120, using the trained distribution network target detection model to process the video image, and identify the target tower in the current video frame; S130, determining the second gimbal angle of the drone at the next moment according to the position information of the target pole tower in the current video frame and the first gimbal angle of the drone at the current moment; determining the flight distance of the drone from the current moment to the next moment according to the position information of the drone at the current moment and the flight speed of the drone; and determining the gimbal attitude change data according to the first gimbal angle and the second gimbal angle; determining the horizontal distance of the drone to the target tower according to the gimbal attitude change data, the flight distance, and the first gimbal angle; determining the position information of the target tower according to the horizontal distance and the drone position information at the current moment; determining the flight trajectory information of the drone, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment according to the position information of the target tower and the drone position information at the current moment, so as to control the drone to fly according to the flight trajectory information, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment; S140, when the drone reaches directly above the target tower, determine the position of a first photographing point according to the drone position information, the heading angle of the drone, and the height of the target tower, and use the position of the first photographing point to control the drone to fly to the first photographing point; S150, when the UAV arrives at the target photographing point, obtain the current view captured by the gimbal camera, process the current view using the distribution network target detection model, and determine the minimum bounding box of each target component according to the processing result; use the minimum bounding box of each target component to respectively determine the position and size of each target component in the current view; for each target component, based on the position of the target component in the current view and the current focal length of the gimbal camera, determine the angular offset of the target component relative to the gimbal of the UAV, based on the current gimbal angle of the UAV and the angular offset of the target component relative to the gimbal of the UAV, determine the gimbal angle of the UAV to photograph the target component, based on the current zoom factor of the gimbal camera and the size of the target component in the current view, determine the zoom factor of the UAV to photograph the target component, and control the gimbal camera to photograph the target component according to the gimbal angle and zoom factor of the UAV to photograph the target component; wherein the target photographing point includes one of the four photographing points; S160, when the drone has captured images of all target components at the current target photographing point, determining the position of the next photographing point according to the drone position information, the heading angle of the drone, and the height of the target tower, and controlling the drone to fly to the next photographing point using the position of the next photographing point; S170, looping through at least one of steps S150 and S160 until the drone has captured images of the target components corresponding to the four photographing points; wherein the four photographing points include a front photographing point, a right photographing point, a rear photographing point, and a left photographing point in sequence; S180, control the UAV to fly to the target safe height and complete the task of photographing the target tower.

2. The method according to claim 1, characterized in that The step of determining the horizontal distance of the drone to the target tower pole according to the gimbal attitude change data, the flight distance, and the first gimbal angle includes: The horizontal distance is determined using the following formula: Wherein, α represents the first gimbal angle, β represents the second gimbal angle, tanβ-tanα represents the gimbal attitude change data, v*dt represents the flight distance, and S represents the horizontal distance.

3. The method according to claim 1, characterized in that Determining the position of the first photo taking point according to the drone position information, the flight direction, and the height of the target tower includes: Use the following formula to determine the location of the first photo point: Among them, (lat1, lon1) represents the position information of the drone, d represents the height of the target tower, heading represents the heading angle of the current drone, and R represents the radius of the earth ellipsoid.

4. The method according to claim 1, characterized in that The method of using the minimum bounding box of each target component to determine the position and size of each target component in the current view includes: For each target component p, the position of the corresponding target component in the current view is determined using the following formula: p xy =min(obb) Among them, p xy (p x ,p y ) represents the position of the target component in the current view, and obb represents the coordinates of the four vertices of the minimum bounding box of the target component; For each target component, the size of the corresponding target component in the current view is determined using the following formula: more h =dis(more) Among them, obb h Indicates the size of the target component in the current view.

5. The method according to claim 1, characterized in that The step of determining the angular offset of the target component relative to the gimbal of the drone based on the position of the target component in the current view and the current focal length of the gimbal camera comprises: delta_α=arctan(d y / fl_px) Among them, P y Indicates the position of the target component in the y-axis direction in the current view, H indicates the height of the current view, dy is the offset of the target component to the center of the image, fl_px indicates the current focal length of the gimbal camera, and delta_α indicates the angular offset of the target component relative to the gimbal of the drone.

6. The method according to claim 1, characterized in that The determining the pan-tilt angle of the drone for photographing the target component based on the current pan-tilt angle of the drone and the angular offset of the target component relative to the pan-tilt of the drone comprises: next_β=delta_α+cur_β Among them, cur_β represents the current gimbal angle of the drone, delta_α represents the angular offset of the target component relative to the drone's gimbal, and next_β represents the gimbal angle of the drone when shooting the target component.

7. The method according to claim 1, characterized in that The method of determining the zoom factor of the target component photographed by the drone based on the current zoom factor of the gimbal camera and the size of the target component in the current view includes: Among them, nextR represents the zoom factor of the drone shooting the target component, and curR represents the current zoom factor of the gimbal camera.

8. A visual perception driven distribution network drone inspection device, characterized in that: include: A data acquisition module, used to acquire video images captured by a pan-tilt camera mounted on a drone, drone position information collected by an RTK module mounted on the drone, and drone attitude information; wherein the drone attitude information includes the pan-tilt angle of the drone; A model processing module is used to process the video image using the trained distribution network target detection model to identify the target tower in the current video frame; A path planning module, for determining the second gimbal angle of the drone at the next moment according to the position information of the target pole tower in the current video frame and the first gimbal angle of the drone at the current moment; determining the flight distance of the drone from the current moment to the next moment according to the position information of the drone at the current moment and the flight speed of the drone; and determining the gimbal attitude change data according to the first gimbal angle and the second gimbal angle; determining the horizontal distance of the drone to the target tower according to the gimbal attitude change data, the flight distance, and the first gimbal angle; determining the position information of the target tower according to the horizontal distance and the drone position information at the current moment; determining the flight trajectory information of the drone, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment according to the position information of the target tower and the drone position information at the current moment, so as to control the drone to fly according to the flight trajectory information, the flight direction of the drone at the next moment, and the attitude information of the drone at the next moment; A photographing point determination module is used to determine the position of a first photographing point according to the position information of the drone, the heading angle of the drone, and the height of the target tower when the drone reaches the top of the target tower, so as to control the drone to fly to the first photographing point by using the position of the first photographing point; A precision photographing module, used for obtaining the current view captured by the gimbal camera when the UAV reaches the target photographing point, processing the current view using the distribution network target detection model, and determining the minimum bounding box of each target component according to the processing result; using the minimum bounding box of each target component to respectively determine the position and size of each target component in the current view; for each target component, based on the position of the target component in the current view and the current focal length of the gimbal camera, determining the angular offset of the target component relative to the gimbal of the UAV, based on the current gimbal angle of the UAV and the angular offset of the target component relative to the gimbal of the UAV, determining the gimbal angle of the UAV for photographing the target component, based on the current zoom factor of the gimbal camera and the size of the target component in the current view, and controlling the gimbal camera to photograph the target component according to the gimbal angle and zoom factor of the UAV for photographing the target component; wherein the target photographing point includes one of the four photographing points; The photographing point determination module is further used to determine the position of the next photographing point according to the position information of the drone, the heading angle of the drone, and the height of the target tower when the drone has taken images of all target components at the current target photographing point, so as to control the drone to fly to the next photographing point by using the position of the next photographing point; The precision photographing module is further used to photograph the target component corresponding to each photographing point until the drone has finished photographing the target components corresponding to the four photographing points; wherein the four photographing points include the front photographing point, the right photographing point, the rear photographing point, and the left photographing point in sequence; The homing module is used to control the drone to fly to the target safe altitude and complete the task of photographing the target tower.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images