A Progressive Target Focusing Method for UAV Gimbal Based on Intelligent Vision Control

Abstract

A progressive target focusing method for a drone gimbal based on intelligent vision control, belonging to the technical fields of target focusing, deep learning, and drone gimbal control; the technical problem to be solved is: to provide an improvement of a progressive target focusing method for a drone gimbal; the technical solution adopted to solve the above technical problem is: obtain a transmission line image sample database; construct a device detection data set and a component defect data set; construct a dual target detection model based on YOLOv5 and SSPNet and perform model training; obtain the image data stream of the real scene of the transmission line through the drone; after the input of the real-time real scene image data stream, automatically call the two detection models at different times and transmit the target information to the progressive target focusing module of the drone gimbal; after obtaining the target frame size, coordinates, and gimbal attitude information, save them, and transmit the target parameters to the core algorithm module for calculation to achieve progressive camera focusing and gimbal control; the present invention is applied to drones.

Description

Technical Field

[0001] A progressive target focusing method for an unmanned aerial vehicle (UAV) gimbal based on intelligent vision control according to the present invention belongs to the technical field of progressive target focusing methods for UAV gimbals. Background Technique

[0002] In recent years, the power grid facilities in China have been continuously expanded and the system has been gradually expanded. The traditional manual inspection method can no longer meet the needs of maintaining the normal and stable operation of the power grid system. In order to improve the efficiency and quality of inspections, power grid companies across the country have gradually carried out research on the inspection operations of transmission lines by UAVs. Although UAV inspections have greatly improved the inspection efficiency, reduced the inspection cost, and reduced the risks of field operations to a large extent, a large amount of image data on transmission line components will be generated during the inspection process. The data is too much to store, and the intelligence level is low and it cannot automatically screen key data. Identifying this data manually is also a huge project. Therefore, proposing a target focusing method for defects of transmission line components based on intelligent vision control has important theoretical significance and application value in the detection of power line equipment.

[0003] For detecting object targets of interest in large-scale images captured by UAVs, it is very challenging due to the large number of object targets and small scales. With the amazing performance improvement of deep learning technology in the fields of computer vision and human-computer interaction, it provides a theoretical and engineering basis for applying artificial intelligence technology to solve practical production problems in industrial scenarios. Deep learning is not just an upgrade of an algorithm, but also a completely new thinking mode. Using deep learning technology greatly improves the accuracy of data acquisition and prediction. However, a large number of experiments show that deep learning models require the support of huge computing power, resulting in difficulty in carrying out real-time abnormal detection and positioning of transmission equipment in the cloud, and the subtle differences in the number and size of targets will greatly affect the model performance and algorithm structure. Therefore, the present invention mainly studies the target detection algorithm for transmission equipment and defects at the front end. Based on the basis of lightweight, real-time, and high-precision, a dual-target detection model for multi-target device components and small-target detail defects is used for intelligent vision control to achieve real-time progressive target focusing of the UAV gimbal for transmission lines.

[0004] When focusing on the target of interest, most of the existing UAV gimbal devices and methods focus and take pictures according to the pre-set or manually controlled positions and angles. This method has a low level of intelligence and still requires a very high labor cost. In addition, another method uses the target recognition and tracking method to focus on and obtain the defective target of interest. However, this method has low stability. When the background environment in the target area is complex and the number of device targets is large, the accuracy of the recognition and tracking focusing method is low, and it is easy to miss the device defect images and generate "dirty data". To solve the above problems, the present invention independently designs a progressive target focusing method for a UAV gimbal, which uses the size components and defect information captured by the dual-model vision control module to gradually expand and contract the camera focal length and adjust the gimbal pitch angle and horizontal yaw angle until the detailed defects are focused and saved. The whole process realizes automation and intelligence. Only by manually controlling the UAV to reach the reference coordinates about 10m away from the target area can all the device defects in the area be captured. Summary of the Invention

[0005] In order to overcome the deficiencies in the prior art, the technical problem to be solved by the present invention is to provide an improvement on the progressive target focusing method for a UAV gimbal based on intelligent vision control.

[0006] To solve the above technical problem, the technical solution adopted by the present invention is as follows: A progressive target focusing method for a UAV gimbal based on intelligent vision control includes the following steps:

[0007] Step 1: Obtain the image sample database of the transmission line.

[0008] Step 2: Label the images in the image sample database in Step 1 to construct the device detection data set and the component defect data set.

[0009] Step 3: Construct a dual target detection model based on YOLOv5 and SSPNet and train the model through the data set to obtain the trained dual detection model.

[0010] Step 4: Obtain the image data stream of the real scene of the transmission line through a UAV with a camera gimbal.

[0011] Step 5: After the real-time real-scene image data stream is input, automatically call the two detection models at different times and transmit the target information to the progressive target focusing module of the UAV gimbal to realize real-time target detection and data feedback.

[0012] After obtaining the target frame size, coordinates and gimbal attitude information, save them, transmit the target parameters to the core algorithm module for calculation, so as to realize progressive camera focusing and gimbal control, and focus on and take pictures of the final target defects and save them in the storage module.

[0013] The device detection dataset in step 2 includes device data for detecting insulators, nuts, and pins on high-voltage transmission lines. The component defect dataset in step 2 includes defect data for detecting missing pins and missing nuts on high-voltage transmission lines.

[0014] The YOLOv5-based object detection model in step 3 mainly realizes real-time multi-object detection of transmission line equipment components. Specifically, it is set as a neural network with an attention mechanism and a deformable convolution module. Through the attention mechanism and the deformable convolution module, during the fine-grained image feature extraction process, it focuses on the detailed information of interest and suppresses other useless information, making the network model pay more attention to the features of the target components of the transmission line.

[0015] The SSPNet-based object detection model in step 3 mainly realizes real-time small-scale object detection of component defects on the transmission line. Specifically, it is set as a neural network with a context attention module, a scale enhancement module, and a scale selection module. Through the context attention module and the scale enhancement module, the network fully considers the context information for stratification and highlights the features of a specific scale at different layers, making the detector focus on objects of a specific scale;

[0016] After that, a scale selection module is introduced to utilize the relationship between adjacent layers to achieve appropriate feature contribution between the deep layer and the shallow layer, thereby avoiding the inconsistency of gradient calculation between different layers.

[0017] Step 5 specifically includes the following steps:

[0018] Step 5.1: The drone flies to a position 10 m away from the target area, and calls the model1 multi-object component detection model, that is, the YOLOv5-based object detection model. Through the real-time image data stream input by the depth camera, it identifies and detects insulator components, marks different insulator components in the area, and saves the quantity m, the corresponding center point coordinates (X c , Y c , S c ), and the initial coordinates of the drone and the initial attitude 1 of the pan-tilt to the storage module for subsequent progressive processing; in the formula, S c represents the depth distance of the insulator from the camera;

[0019] Step 5.2: Return to the initial coordinates and the pan-tilt attitude 1. By comparing the sizes of S c of each insulator in the area, select the insulator component with the closest distance to the drone for focusing operation, and store the pan-tilt attitude 2 of the drone at this time to the storage module.

[0020] According to the position area information of the target object, control the angle of the pan-tilt head, align the center of the field of view with the specified target, and use geometric relationships to deduce the angle between the position of the target object and the center point of the screen;

[0021] Among them, the calculation formula for the center point position of the specified target is:

[0022] The calculation formula for the yaw angle of the specified target is:

[0023] The calculation formula for the pitch angle of the specified target is:

[0024] In the above formula: X c 、Y c represent the center point position information of the detected target object, W and H represent the width and height of the picture, x, y, w, and h represent the position and size of the target box, aov represents the camera view angle, the horizontal yaw angle of the UAV pan-tilt head fine-tuning is α, and the pitch angle is β;

[0025] Step 5.3: The UAV pan-tilt head automatically sides to the unmarked end of the insulator and marks this end. Detect whether the insulator is arranged longitudinally or horizontally through the algorithm function. If it is horizontal, automatically adjust the pan-tilt head to the left or right, and the horizontal yaw angle is α 2 , if it is longitudinal, automatically adjust the pan-tilt head up or down, and the pitch angle is β 2 , then detect whether both ends of the insulator component are marked. If so, delete the insulator information and return to Step 5.2. If not, proceed to Step 5.4;

[0026] Step 5.4: Dynamically adjust the camera focal length, and set the threshold condition to be able to normally detect the detailed components such as nuts and pins, and the proportion of the detailed components of nuts and pins in the picture is appropriate;

[0027] Calculate the correlation coefficient between the obtained target box and the aspect ratio of the picture length and width Realize automatic focal length adjustment;

[0028] After that, calculate the center point coordinates of the detailed components and focus on this center point. The method is the same as in Step 5.2. After focusing, call the Model2 small-scale target defect detection model, that is, the target detection model based on SSPNet, to detect the defects near the detailed components and judge whether there are defects. If not, return to Step 5.3. If so, proceed to Step 5.5;

[0029] Step 5.5: Focus on and take pictures of the defect center points gradually determined by the progressive method, save the images to the storage module, and at the same time return to Step 5.3 for marking and mark detection.

[0030] The beneficial effects of the present invention compared with the prior art are as follows: Compared with the current traditional UAV gimbal focusing algorithms, the present invention combines deep learning technology and proposes a progressive target of interest focusing method, breaking the conventional thinking, improving the automation and intelligence of UAV operations, combining the target feature information of multiple targets with different scales in the spatial area, and automatically detecting and collecting adjacent defects such as insulators within the range through relevant algorithms. The advantages of the present invention are mainly reflected in the following two aspects:

[0031] (1) Precise identification and detection. The method proposed by the present invention can solve the two major detection problems of multiple targets and small scales in complex environments through the dual target detection models trained by different network structures. At the same time, it switches repeatedly according to the process of the designed method, controls the gimbal to progressively detect components such as insulators and target defects, and can efficiently and real-time realize defect identification, detection and collection;

[0032] (2) Progressive target focusing and collection. The present invention no longer uses traditional manual or semi-manual methods for UAV gimbal target focusing and collection, but adopts an innovative method of progressive target focusing and collection under intelligent vision control. Through the target feature information transmitted in real time by the vision control module, the information is calculated from different angles to control the UAV gimbal. Such a method enables the camera to gradually focus on the target defects of interest, and thus perform full-automatic collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described below with reference to the drawings:

[0034] Figure 1 is the flow chart of the method of the present invention;

[0035] Figure 2 is the flow chart of the progressive target focusing method of the present invention;

[0036] Figure 3 is the pseudo-code diagram of the component center point focusing calculation of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0037] As Figures 1 to 3As shown, the present invention provides a progressive target focusing method for the UAV gimbal based on intelligent visual control. Due to the complex environment of the high-altitude power transmission line, the large number of equipment component targets, and the small scale of the target defect, the existing deep learning method cannot meet the two detection requirements of multi-target and small scale while ensuring real-time high precision. This method first obtains the power transmission line image sample database, and then annotates and constructs the corresponding data set. Based on the core ideas of YOLOv5 and SSPNet, the powerful modeling and fitting capabilities of the neural network are fully utilized to construct a dual model for target detection. After the real-time real scene image data stream is input, the two models are automatically called in time-sharing and the target information is transmitted to the progressive target focusing module of the UAV gimbal to realize real-time target detection and data feedback. After obtaining the target frame size, coordinates, gimbal posture and other information, it is saved and transferred to the core algorithm for calculation, thereby realizing progressive camera focus and gimbal control, and the final target defect is focused and photographed, and saved in the storage module. The experimental results show that the dual target detection model deployed in the front-end embedded device can quickly and accurately identify and detect equipment components and target defects. The intelligent vision module obtains data and transmits it to the progressive target focusing module, realizing the focusing and photo storage of the defective image of the power transmission line equipment. The method proposed in the present invention has good performance and promotion and application prospects for focusing and acquiring defective images of high-altitude power transmission line equipment, and provides a typical example for the focus of UAV gimbal targets and the detection and acquisition of transmission line defects. The overall flow chart of the method is shown in the figure. Figure 1 shown.

[0038] The core steps of the method of the present invention are as follows:

[0039] Step 1: Design an intelligent visual control module based on deep learning and train to obtain a dual model for target detection. The present invention takes photos of various target categories in real power transmission line scenes, compiles a sample database, and then selects images of specified components and corresponding defect targets for annotation and constructs a dual model dataset, providing good data support and guarantee for subsequent model training and reasoning. The dataset contains 3 types of components such as insulators and 2 types of equipment defects, as shown in Table 1:

[0040] Table 1 Categories and numbers of dual-model datasets for target detection

[0041]

[0042] Then, the structure of the deep learning target detection network suitable for multi-target and small-scale in complex environments was improved, and the dual target detection model was trained through parameter optimization, as follows:

[0043] Model 1: A real-time multi-object detection model for transmission line components based on YOLOv5 (muti-YOLOv5—Model1). Based on the core idea of YOLOv5, in order to reduce the misdetection of similar objects and speed up the inference speed, an attention mechanism and a deformable convolution module are introduced. During the fine-grained image feature extraction process, the model focuses on the detailed information of interest and suppresses other useless information, enabling the network model to pay more attention to the features of transmission line target components, thereby achieving the purpose of improving the model performance and reducing the misdetection rate, and realizing the effect of high-precision real-time multi-object detection.

[0044] Model 2: A real-time small-scale object detection model for line component defects based on SSPNet (tiny-SSPNet—Model2). Most existing detection methods use feature pyramid networks to enrich shallow features by combining deep context features. This method introduces a context attention module and a scale enhancement module based on the SSPNet idea, enabling the network to fully consider context information for layering and highlighting features at specific scales in different layers, so that the detector focuses on objects at specific scales. Then, a scale selection module is introduced to utilize the relationship between adjacent layers to achieve appropriate feature contribution between the deep and shallow layers, thereby avoiding inconsistent gradient calculations between different layers and realizing high-precision real-time detection of small-scale defects on transmission lines.

[0045] Step 2: Use the data obtained in real time in Step 1 to design a progressive target focusing method to control the pan-tilt head. The model is called in stages through the algorithm to obtain the target feature information transmitted by the intelligent vision control module. Then, the initial focal length is adjusted by calculation from far to near, enabling clear detection of detail devices such as insulators. Subsequently, focus and pan-tilt head control are performed on both ends of the device respectively to identify, detect, and collect device defects in real time and accurately. After the detection is completed, the pan-tilt head returns to the reference pose to start the detection of the next device. The entire focusing and acquisition method conforms to the characteristics of automation and progression, and can almost fully automatically focus on and capture target defects from far to near and from the edge to the center. At the same time, after testing, this method can complete the focusing and image acquisition tasks with high standards in real scenarios. The specific process is as Figure 2 shown.

[0046] Step 2.1: The UAV flies to a position 10 m away from the target area, calls the model1 multi-object component detection model, identifies and detects insulator components through the real-time image data stream transmitted by the depth camera, marks different insulator components in the area, and saves the quantity m, the corresponding center point coordinates (X c , Y c , S c ), and the initial coordinates of the UAV and the initial pose 1 of the pan-tilt head to the storage module for subsequent progressive processing. In the formula, S c represents the depth distance of the insulator from the camera.

[0047] Step 2.2: Return to the initial coordinates and the pan-tilt attitude 1. By comparing the sizes of each insulator S within the area, select the insulator component with the shortest distance to the UAV for focusing. According to the position area information of the target object, control the pan-tilt angle to align the center of the field of view with the specified target. Use geometric relationships to deduce the angle between the position of the target object and the center point of the screen: c Position of the center point of the specified target:

[0048] Yaw angle:

[0049] Pitch angle:

[0050] Where X

[0051] , Y c represent the position information of the center point of the detected target object, W and H represent the width and height of the picture, x, y, w, and h represent the position and size of the target box, aov represents the camera view angle, the horizontal yaw angle of the UAV pan-tilt fine-tuning is α, and the pitch is β. After focusing, store the pan-tilt attitude 2 of the UAV at this time in the storage module. c Step 2.3: The UAV pan-tilt automatically tilts towards the unmarked end of the insulator and marks this end. Detect whether the insulator arrangement is vertical or horizontal through an algorithm function. If it is horizontal, automatically adjust the pan-tilt to the left or right, and the horizontal yaw angle is α

[0052] , if it is vertical, automatically adjust the pan-tilt up or down, and the pitch angle is β 2 . Subsequently, detect whether both ends of the insulator component are marked. If so, delete the insulator information and return to Step 2.2. If not, proceed to Step 2.4. 2 Yaw angle:

[0053]

[0054] Pitch angle:

[0055] Step 2.4: Dynamically adjust the camera focal length. Set the threshold condition to be able to detect details such as nuts normally, and the proportion of details such as nuts in the picture is appropriate. The common adjustment range of the camera focal length multiple ∈ [2x, 4x, 6x, 8x, 10x], and calculate the correlation coefficient λ between the obtained target box and the picture length-width ratio ∈ [20, 25, 30, 35, 40]. The arrays correspond to each other to achieve automatic focal length adjustment.

[0056]

[0057] ​After that, store the coordinate data of each detailed component in a list structure, calculate the center point coordinates of the detailed component, and focus on this center point. The method is the same as step 2.2. After focusing, call the Model2 small-scale target defect detection model to detect the defects near the detailed component and determine whether there are defects. If not, return to step 2.3. If so, proceed to step 2.5. The pseudo-code for central focusing on the component is as Figure 3 shown.

[0058] Step 2.5: Focus on and take a picture of the defect center point gradually determined by the progressive method, save the image to the storage module, and at the same time return to step 2.3 for marking and marking detection.

[0059] The present invention discloses a progressive target focusing method for an unmanned aerial vehicle (UAV) gimbal based on intelligent vision control. An unmanned aerial vehicle equipped with a camera gimbal is arranged for image acquisition. A lightweight multi-target and small-scale depth target detection dual model is used to analyze and process transmission line components and defects. While identifying and detecting, the target feature information is transmitted to the progressive target focusing module of the UAV gimbal. Through the calculation of information such as target features, the angle of the UAV gimbal is automatically adjusted progressively to gradually focus it on the target defects near different insulators in the area. Subsequently, the images that meet the specified threshold are taken and saved to obtain defect images. The disclosed method can meet the practical requirements in terms of identification and detection accuracy and inference speed, and realizes the application of the UAV in the high-altitude transmission line scenario.

[0060] Regarding the specific structure of the present invention, it should be noted that the connection relationships between the various component modules adopted by the present invention are determined and achievable. Except for the special descriptions in the embodiments, the specific connection relationships can bring corresponding technical effects and, on the premise of not relying on the execution of corresponding software programs, solve the technical problems proposed by the present invention. The models and connection methods of the components, modules, and specific components in the present invention, except for the specific descriptions, all belong to the prior art such as publicly disclosed patents, publicly disclosed journal papers, or common general knowledge that those skilled in the art can obtain before the filing date, and need not be elaborated. This makes the technical solution provided in this case clear, complete, and achievable, and can reproduce or obtain the corresponding physical product according to this technical means.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A progressive target focusing method for an unmanned aerial vehicle (UAV) gimbal based on intelligent vision control, characterized in that: it includes the following steps: Step 1: Obtain an image sample database of the transmission line; Step 2: Annotate the images in the image sample database in Step 1 to construct an equipment detection data set and a component defect data set; Step 3: Construct a dual target detection model based on YOLOv5 and SSPNet and train the model through the data set to obtain a trained dual detection model; Step 4: Obtain an image data stream of the real scene of the transmission line through a UAV with a camera gimbal; Step 5: After the input of the real-time real scene image data stream, automatically call the two detection models at different times and transmit the target information to the progressive target focusing module of the UAV gimbal to achieve real-time target detection and data feedback; After obtaining the target bounding box size, coordinates and gimbal attitude information, save them, transmit the target parameters to the core algorithm module for calculation, so as to achieve progressive camera focusing and gimbal control, and focus on and photograph the final target defect, and save it in the storage module; The specific steps of Step 5 include the following steps: Step 5.1: The drone flies to a position 10 m away from the target area, calls the model1 multi-target component detection model, i.e., the object detection model based on YOLOv5, and identifies and detects insulator components through the real-time image data stream input by the depth camera, marks different insulator components in the area, and saves the quantity m, the corresponding center point coordinates (X c , Y c , S c ), the initial coordinates of the drone, and the initial attitude of the pan-tilt to the storage module to prepare for subsequent progressive processing; in the formula, S c represents the depth distance of the insulator from the camera; Step 5.2: Return to the initial coordinates and the pan-tilt attitude 1. By comparing the sizes of each insulator S within the area, select the insulator component with the shortest distance to the UAV for focusing operation. After focusing, store the pan-tilt attitude 2 of the UAV at this time in the storage module; c After that, store the pan-tilt attitude 2 of the UAV at this time in the storage module; According to the position area information of the target object, control the gimbal angle to align the center of the field of view with the specified target, and use geometric relationships to deduce the angle between the position of the target object and the center point of the screen; The calculation formula for specifying the position of the target center point is as follows: The yaw angle calculation formula for the specified target is as follows: The calculation formula for the pitch angle of the specified target is as follows: In the above formula: X c , Y c represent the center point position information of the detected target object, W and H represent the width and height of the picture, x, y, w, and h represent the position and size of the target box, aov represents the camera view angle, the horizontal yaw angle of the UAV gimbal fine-tuning is α, and the pitch angle is β; Step 5.3: The drone gimbal automatically tilts towards the unmarked end of the insulator and marks this end. Detect whether the insulator is arranged longitudinally or transversely through an algorithm function. If it is transverse, the gimbal is automatically adjusted left or right, and the horizontal yaw angle is α 2 , if it is longitudinal, the gimbal is automatically adjusted up or down, and the pitch angle is β 2 , then detect whether both ends of the insulator component are marked. If so, delete the insulator information and return to Step 5.

2. If not, proceed to Step 5.4; Step 5.4: Dynamically adjust the camera focal length, and set the threshold condition to be able to normally detect the detailed components such as nuts and pins, and the proportion of the detailed components of nuts and pins in the picture is appropriate; Calculated correlation coefficient between the target bounding box and the aspect ratio of the image Implement automatic focus adjustment; After that, calculate the center point coordinates of the detailed components and focus on this center point. The method is the same as Step 5.

2. After focusing, call the Model2 small-scale target defect detection model, that is, the target detection model based on SSPNet, to detect the defects near the detailed components and judge whether there are defects. If not, return to Step 5.

3. If so, perform Step 5.5; Step 5.5: Focus on and photograph the defect center points gradually determined by the progressive method, save the images to the storage module, and at the same time return to Step 5.3 for marking and marking detection.

2. A progressive target focusing method for an unmanned aerial vehicle (UAV) gimbal based on intelligent vision control according to claim 1, characterized in that: The equipment detection data set in Step 2 includes equipment data for detecting insulators, nuts, and pins of high-voltage transmission lines, and the component defect data set in Step 2 includes defect data for detecting missing pins and missing nuts of high-voltage transmission lines.

3. A progressive target focusing method for an unmanned aerial vehicle (UAV) gimbal based on intelligent vision control according to claim 1, characterized in that: The target detection model based on YOLOv5 in Step 3 mainly realizes real-time multi-target detection of transmission line equipment components. Specifically, it is set as a neural network with an attention mechanism and a deformable convolution module. Through the attention mechanism and the deformable convolution module, during the process of fine-grained image feature extraction, pay attention to the detailed information of interest and suppress other useless information, so that the network model pays more attention to the features of the transmission line target components.

4. A progressive target focusing method for an unmanned aerial vehicle gimbal based on intelligent vision control according to claim 1, characterized in that: The target detection model based on SSPNet in step three mainly realizes real-time small-scale target detection of component defects on transmission lines, and is specifically set as a neural network with a context attention module, a scale enhancement module, and a scale selection module. Through the context attention module and the scale enhancement module, the network fully considers context information for layering, and highlights features of specific scales at different layers, enabling the detector to focus on objects of specific scales; After that, a scale selection module is introduced to utilize the relationship between adjacent layers to achieve appropriate feature contribution between the deep layer and the shallow layer, thereby avoiding inconsistent gradient calculations between different layers.

Images