Unmanned aerial vehicle pure vision side autonomous line simulation method and device, equipment and medium

Abstract

The invention provides an unmanned aerial vehicle pure vision side autonomous line imitation method and device, equipment and a medium. The method comprises the steps of performing semantic segmentation on a real-time environment image collected by a camera, performing instance segmentation on a semantic segmentation result, and constructing a power transmission line mask instance; performing tracking processing on each power transmission line mask instance by using a target tracking algorithm, updating the state of each power transmission line, and predicting the position and speed of the next frame of unmanned aerial vehicle; the target tracking algorithm comprises a closed-loop finite-state machine, an interactive multi-model and a multi-stage cascade matching algorithm; adjusting the position and speed of the next frame of unmanned aerial vehicle according to the plane constraint; planning a flight path of the unmanned aerial vehicle by using the position and the speed of the unmanned aerial vehicle of the current frame and the position and the speed of the unmanned aerial vehicle of the next frame, and generating a plurality of flight instructions of the unmanned aerial vehicle from the current frame to the next frame; and circularly executing the steps until the tower head insulator is identified by using the obtained real-time environment image, and completing the current tower pole line simulation task.

Description

Technical Field

[0001] The present invention relates to the field of data processing, in particular to the field of electric power line imitation technology, and discloses a method and device, equipment, and medium for autonomous line imitation by the side of a drone based on pure vision. Background Art

[0002] Drone visual edge tracking (also known as edge following or boundary tracking) is a technology that allows drones to fly along a pre-set path or along the edge of a specific object. This technology is commonly used in the fields of electricity, agriculture, infrastructure (such as power lines, pipelines) inspection, photography and mapping.

[0003] The current mainstream solution is the radar-based drone line-following solution, which has the following three technical defects:

[0004] 1. High price: High-quality lidar sensors are expensive, which increases the overall cost of the drone system, which limits its promotion and use in certain application areas.

[0005] 2. High power consumption and endurance issues: LiDAR consumes a lot of power when working, which poses a challenge to the battery life of the drone, thus affecting the flight time and operating range. Since the LiDAR equipment itself is relatively heavy, it will take up additional payload of the drone, reducing the space for other mission equipment or reducing the overall endurance.

[0006] 3. High data processing complexity: The amount of point cloud data generated by LiDAR is huge. Real-time processing of such a large data set requires powerful computing power. It is difficult for onboard computers to meet such a large computing resource. In order to extract useful information from the point cloud and perform accurate target recognition and tracking, complex algorithm support is usually required, which not only increases the difficulty of development, but may also cause delay problems.

[0007] In view of the above problems, there is currently a vision-based drone line-following solution, but this solution has the following defects:

[0008] 1. Difficulty in wire extraction: Power wires are usually thin and long metal wires, which appear as thin lines in the image, which brings challenges to edge detection and feature extraction. In addition, different lighting conditions, including direct sunlight, shadows and insufficient lighting in cloudy environments, and various environmental backgrounds, will affect image quality and make edge recognition difficult.

[0009] 2. Difficulty in following the line: Low-resolution cameras may not provide enough details to clearly see the edge line; and although wide-angle lenses can cover a wider field of view, they will make the distant edge lines appear smaller, which is not conducive to accurate identification. In order to maintain a stable flight state, the attitude of the drone (such as pitch, roll and yaw, acceleration, speed) needs to be accurately controlled. This is particularly difficult in complex environments (large height difference transmission line slope, large slope changes). The drone needs to be accurately, stably, and quickly controlled. Otherwise, the transmission line can easily fly out of the camera's field of view, resulting in failure of line imitation. During the drone line imitation process, the drone speed and attitude must be dynamically adjusted according to the surrounding environment to ensure a safe distance and avoid collisions.

[0010] Due to the above defects, the visual UAV line-following scheme has problems of low accuracy, poor adaptability and poor robustness. Summary of the invention

[0011] The present disclosure at least provides a method and device, equipment, and medium for autonomous line-following of the side of a drone purely by vision, so as to overcome at least one technical problem existing in the solution based on vision line-following.

[0012] According to one aspect of the present disclosure, a method for autonomously imitating a line on the side of a drone using pure vision is provided, comprising:

[0013] S110, obtaining a real-time environment image captured by a camera carried by the UAV;

[0014] S120, performing semantic segmentation on the real-time environment image to obtain a power line mask; performing instance segmentation on the power line mask, and constructing a power line mask instance according to the instance segmentation result;

[0015] S130, using a target tracking algorithm to track each transmission line mask instance respectively, and updating the state of the corresponding transmission line according to the result of the tracking process, and predicting the position and speed of the drone in the next frame; wherein the target tracking algorithm includes a closed-loop finite state machine, an interactive multi-model, and a multi-level cascade matching algorithm;

[0016] S140, obtaining the plane constraint of the UAV flight, and adjusting the position and speed of the UAV in the next frame according to the plane constraint;

[0017] S150, using the position of the drone in the current frame, the speed of the drone in the current frame, the position of the drone in the next frame, and the speed of the drone in the next frame, to plan the flight path of the drone, and generate multiple flight instructions for the drone between the current frame and the next frame; the flight instructions include speed instructions and path instructions;

[0018] S160, looping through steps S110-150 until the tower head insulator is identified by using the acquired real-time environment image.

[0019] In a possible implementation, the tracking process of each transmission line mask instance is performed using a target tracking algorithm, and the state of the corresponding transmission line is updated according to the result of the tracking process, including:

[0020] Acquire a closed-loop finite state machine; wherein the closed-loop finite state machine includes a current frame trajectory set, an existing trajectory set, and a trajectory state set for storing the state of the existing trajectory;

[0021] Determine the center point of each existing trajectory according to the existing trajectory set; and determine the line segment endpoints corresponding to each transmission line according to each transmission line mask instance;

[0022] A cost matrix is ​​constructed according to the center points of the existing trajectories and the line segment endpoints corresponding to each transmission line; the cost function of the cost matrix is: the distance from a point to a straight line, wherein the point is the center point of each existing trajectory and the straight line is the straight line formed by the line segment endpoints;

[0023] Determine existing trajectories matching each transmission line based on the cost matrix using a multi-stage cascade matching algorithm;

[0024] The status of the existing trajectory that is not matched to the transmission line in the trajectory state set is updated to lost; for the transmission line that is not matched to the existing trajectory, a trajectory is created according to the transmission line mask instance, and the created trajectory is stored in the existing trajectory set, and the status of the created trajectory is set to new, and updated into the trajectory state set; for the transmission line and the existing trajectory that are successfully matched, the position and center point of the corresponding existing trajectory are updated according to the transmission line mask instance, and the status of the existing trajectory in the trajectory state set is updated to active.

[0025] In a possible implementation, the UAV pure vision side autonomous line imitation method further includes:

[0026] For an existing trajectory whose state is lost in the trajectory state set, if the duration of the existing trajectory being in the lost state exceeds a preset duration, the existing trajectory is removed from the existing trajectory set, and the state of the existing trajectory is removed from the trajectory state set.

[0027] In a possible implementation, predicting the position and speed of the drone in the next frame includes:

[0028] The sub-models of each motion mode in the interactive multi-model are used to process the transmission line mask instances respectively, and the predicted position and predicted speed corresponding to each motion mode are obtained;

[0029] The predicted positions corresponding to each motion mode are weighted summed to obtain the position of the drone in the next frame;

[0030] The predicted speeds corresponding to each motion mode are weighted summed to obtain the speed of the drone in the next frame;

[0031] The motion modes include a motion mode corresponding to stillness, a motion mode corresponding to uniform linear motion, and a motion mode corresponding to uniformly accelerated linear motion.

[0032] In a possible implementation, the two-dimensional plane corresponding to the plane constraint is parallel to the plane where the transmission line is located.

[0033] In a possible implementation, the method of using the position of the drone in the current frame, the speed of the drone in the current frame, the position of the drone in the next frame, and the speed of the drone in the next frame to plan the flight path of the drone, and generating multiple flight instructions for the drone between the current frame and the next frame, includes:

[0034] The Hermite spline interpolation method is used to generate a flight path based on the current frame drone position, the current frame drone speed, the next frame drone position, and the next frame drone speed;

[0035] Through the feedback mechanism of the PID controller, when the UAV flies along the flight path, a path instruction for adjusting the position of the UAV and a speed instruction for adjusting the speed and acceleration are generated.

[0036] In a possible implementation manner, performing instance segmentation on the power line mask includes:

[0037] The transmission line mask is instance segmented by searching for connected domains.

[0038] According to another aspect of the present disclosure, a purely visual side autonomous line-tracking device for a drone is provided, comprising:

[0039] An image acquisition module is used to acquire real-time environmental images collected by a camera carried on the drone;

[0040] A segmentation module is used to perform semantic segmentation on the real-time environment image to obtain a power line mask; perform instance segmentation on the power line mask, and construct a power line mask instance according to the instance segmentation result;

[0041] A target tracking module is used to track each transmission line mask instance using a target tracking algorithm, and update the state of the corresponding transmission line according to the result of the tracking process, and predict the position and speed of the drone in the next frame; wherein the target tracking algorithm includes a closed-loop finite state machine, an interactive multi-model, and a multi-level cascade matching algorithm;

[0042] The UAV control module is used to obtain the plane constraints of the UAV flight and adjust the position and speed of the UAV in the next frame according to the plane constraints;

[0043] The drone control module is further used to plan the drone flight path by using the position of the drone in the current frame, the speed of the drone in the current frame, the position of the drone in the next frame, and the speed of the drone in the next frame, and to generate multiple flight instructions for the drone between the current frame and the next frame; the flight instructions include speed instructions and path instructions;

[0044] The loop control module is used to control the image acquisition module, the segmentation module, the target tracking module, and the UAV control module to loop through corresponding steps until the tower head insulator is obtained by using the acquired real-time environmental image recognition.

[0045] According to another aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0046] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned methods is implemented.

[0047] The present invention discloses a method and device for autonomous line simulation of the side edge of a drone using pure vision. First, a real-time environment image captured by a camera mounted on the drone is obtained. Then, the real-time environment image is semantically segmented to obtain a transmission line mask. The transmission line mask is instance segmented, and a transmission line mask instance is constructed according to the instance segmentation result. Then, each transmission line mask instance is tracked and processed by a target tracking algorithm, and the state of the corresponding transmission line is updated according to the result of the tracking process, and the position and speed of the drone in the next frame are predicted. The target tracking algorithm includes a closed-loop finite state machine, an interactive multi-model, and a multi-level cascade matching algorithm. Then, the plane constraint of the drone flight is obtained, and the position and speed of the drone in the next frame are adjusted according to the plane constraint. Then, the drone flight path is planned by using the position of the drone in the current frame, the speed of the drone in the current frame, the position of the drone in the next frame, and the speed of the drone in the next frame, and a plurality of flight instructions of the drone between the current frame and the next frame are generated. The flight instructions include speed instructions and path instructions. The above steps are executed repeatedly until the tower head insulator is identified by using the acquired real-time environment image, and the current tower line simulation task is completed. The above-mentioned scheme disclosed in the present invention improves the stability, accuracy and adaptability of the UAV's side visual line-following task through visual perception of power transmission lines, real-time path normalization and tracking of UAVs, and attitude and speed control scheme of UAVs. It can enable the UAV to complete the side line-following task stably, accurately, safely and quickly, and can be applied to various UAV models, single or multiple conductors, large or small height differences and other scenarios.

[0048] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0050] Figure 1 is a flow chart of the purely visual side autonomous line-mimicking method for a drone disclosed herein;

[0051] Figure 2A is a schematic diagram of a closed-loop finite state machine according to an embodiment of the present disclosure;

[0052] Figure 2B is a schematic diagram of plane constraints according to an embodiment of the present disclosure;

[0053] Figure 3 is a schematic diagram of Hermite according to an embodiment of the present disclosure;

[0054] Figure 4is a structural schematic diagram of a purely visual side autonomous line-following device for a drone disclosed in the present invention;

[0055] Figure 5 is a schematic structural diagram of an electronic device according to the present disclosure. DETAILED DESCRIPTION

[0056] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0057] The present invention aims to solve the problems of low accuracy, poor adaptability and poor robustness of visual UAV line simulation schemes, and provides a UAV pure visual side autonomous line simulation method and device, equipment and medium. The present invention performs semantic segmentation on the real-time environment image collected by the camera, performs instance segmentation on the result of the semantic segmentation, and constructs a transmission line mask instance; uses a target tracking algorithm to track and process each transmission line mask instance respectively, updates the state of each transmission line, and predicts the position and speed of the UAV in the next frame; the target tracking algorithm includes a closed-loop finite state machine, an interactive multi-model, and a multi-level cascade matching algorithm; then adjusts the position and speed of the UAV in the next frame according to the plane constraints; uses the position and speed of the UAV in the current frame and the position and speed of the UAV in the next frame to plan the flight path of the UAV, and generates multiple flight instructions for the UAV between the current frame and the next frame; and executes the above steps in a loop until the tower head insulator is identified by using the acquired real-time environment image, and the current tower line simulation task is completed. The above-mentioned scheme disclosed in the present invention improves the stability, accuracy and adaptability of the UAV's side visual line-following task through visual perception of power transmission lines, real-time path normalization and tracking of UAVs, and attitude and speed control scheme of UAVs. It can enable the UAV to complete the side line-following task stably, accurately, safely and quickly, and can be applied to various UAV models, single or multiple conductors, large or small height differences and other scenarios.

[0058] The technical solution of the present disclosure is described below through specific embodiments.

[0059] like Figure 1 As shown, it is a flow chart of a method for autonomous line imitation of the side of a drone based on pure vision in this embodiment. The execution subject of this embodiment is a computing device or component with data processing capability. Specifically, the method of this embodiment may include the following steps:

[0060] S110: Acquire a real-time environment image captured by a camera carried by the UAV.

[0061] In this step, the drone's onboard camera is used to observe and perceive the surrounding environment and collect real-time environmental images.

[0062] S120, performing semantic segmentation on the real-time environment image to obtain a power line mask; performing instance segmentation on the power line mask, and constructing a power line mask instance according to the instance segmentation result.

[0063] This step converts semantic segmentation into instance segmentation through a post-processing algorithm: one mask instance for each transmission line, and each mask instance can be numbered 0, 1, 2... In this step, corresponding straight line instances (one or more) can be constructed based on the transmission line mask instance, and straight line instances can also be constructed when the straight lines corresponding to the transmission line mask instances are needed in subsequent steps.

[0064] There are two reasons why semantic segmentation is used instead of instance segmentation in this step: first, instance segmentation is not robust enough for "thin" targets such as wires; second, semantic segmentation is usually faster and can more easily meet the real-time (>=10fps) perception requirements of drones.

[0065] In this step, the method of converting the semantic segmentation of transmission lines into instance segmentation is to find connected domains. After being converted into instance segmentation, some post-processing is required to filter out "false wire instances" that do not meet the requirements: line segments with too few points, too short horizontal distances, and abnormal length-to-width ratios (for example, the vertical span exceeds the horizontal span).

[0066] Finding connected components: It is a fundamental problem in image processing and computer vision, usually used for binary images. A connected component refers to an area in an image that consists of pixels with the same attributes (for example, the same grayscale value or color), and these pixels are connected to each other under a certain defined adjacency relationship.

[0067] For binary images, there are usually two definitions: 4-connected and 8-connected:

[0068] 4-Connected: A pixel is connected to its adjacent pixels in four directions: up, down, left, and right.

[0069] 8-connectivity: In addition to the four directions of 4-connectivity, it also includes four pixels in the diagonal direction.

[0070] There are many ways to implement the algorithm for finding connected domains, including but not limited to depth-first search (DFS), breadth-first search (BFS), union-find, etc. This method uses depth-first search DFS.

[0071] S130. Use a target tracking algorithm to track each transmission line mask instance separately, and update the state of the corresponding transmission line according to the result of the tracking process, and predict the position and speed of the drone in the next frame; wherein the target tracking algorithm includes a closed-loop finite state machine, an interactive multi-model, and a multi-level cascade matching algorithm.

[0072] In this step, each transmission line has a tracking target ID number, and the tracking frame rate is 10fps (10HZ).

[0073] Closed loop finite state machine is CLFSM: close loop finite state machine; interactive multiple model is IMM: Interactive multiple model; multi-level cascade matching algorithm is MLCM: Multi-level cascadematching.

[0074] like Figure 2A As shown, the closed-loop finite state machine includes a current frame trajectory set, an existing trajectory set, and a trajectory state set for storing the state of the existing trajectory. The trajectory state set includes: new trajectory (New), activated trajectory (Activated), lost trajectory (Lost), and deleted trajectory (Removed). The current frame trajectory set includes: the current frame unconfirmed (Unconfirmed) list, the current frame tracked (Tracked) list, the current frame activated (Activated) list, the current frame lost (Lost) list, the current frame deleted (Removed) list, etc. The existing trajectory set includes the existing trajectory tracked list (Tracked), the existing trajectory lost list (Lost), and the current trajectory deleted list (Removed).

[0075] This step can switch and update the finite state between the current frame tracking lists, and then interact with the existing trajectory tracking list to complete the closed-loop lifecycle management of the tracking target. The advantages of closed-loop lifecycle management based on finite state machines are: from the perspective of the program, no memory leaks will occur, and it is convenient to debug and track the status of each tracking target; from the perspective of the algorithm, the temporarily lost tracking targets can be recalled with the highest probability to ensure the tracking success rate.

[0076] S140: Acquire the plane constraints of the UAV flight, and adjust the position and speed of the UAV in the next frame according to the plane constraints.

[0077] The two-dimensional plane corresponding to the plane constraint is parallel to the plane where the transmission line is located. There are two main benefits to using plane constraints, or in other words, two problems are solved: 1. Plane constraints can constrain the movement of drones to a 2D spatial plane, ensuring the safety of drones and transmission lines: drones move within a predefined plane, and there is no risk of collision with transmission lines or towers; 2. The movement of drones is simplified from 3D to 2D, which reduces the difficulty of control, reduces vibration and noise, and the direction of the transmission line can be used to guide the movement direction of the drone.

[0078] like Figure 2B As shown, there are two power towers AB in space, and there is a parabolic transmission line between the towers. There is a spatial constraint plane parallel to the straight line connecting the towers. The drone performs line-simulating motion on this constraint plane. This plane is determined by the pilot through the drone marking points before starting the line simulation. A point is marked at the starting point of the line simulation (near A, a certain distance of 10 to 15 meters from the tower A and the conductor), and at the end point of the line simulation (near B, a certain distance of 10 to 15 meters from the tower B and the conductor). Any of the two points can be translated along the z-axis direction (the Z-axis direction is perpendicular to the straight line formed by points A and B) for a certain distance to obtain a third point C. Based on these three points, a unique vertical plane area can be determined. The formed plane is kept parallel to the transmission line as much as possible, but it does not need to be completely parallel. During the line simulation, the drone does not collide with the transmission facilities. As shown in the figure, Figure 2B As shown, the plane constraint is the range of the drone's operation, which constrains the 3D motion to the motion of the 2D plane, and the displacement and speed in the y direction remain 0.

[0079] S150, using the position of the drone in the current frame, the speed of the drone in the current frame, the position of the drone in the next frame, and the speed of the drone in the next frame, to plan the flight path of the drone, and generate multiple flight instructions for the drone between the current frame and the next frame; the flight instructions include speed instructions and path instructions.

[0080] This step uses the Hermite spline interpolation method to generate a flight path based on the position of the drone in the current frame, the speed of the drone in the current frame, the position of the drone in the next frame, and the speed of the drone in the next frame; then, through the feedback mechanism of the PID controller, while the drone is flying along the flight path, path instructions for adjusting the position of the drone, and speed instructions for adjusting the speed and acceleration are generated.

[0081] In some embodiments, Figure 3As shown, multiple interpolation commands, such as 10 speed commands, are required between two frames, where: Start Position represents the position of the drone in the current frame, End Position represents the position of the drone in the next frame (end position), Position represents the interpolated drone position, Start Velocity represents the vector of the drone's velocity in the current frame, End Velocity represents the vector of the drone's velocity in the next frame, and Acceleration represents the acceleration direction vector that the drone needs to reach at each position. The Hermite interpolation polynomial is used to perform path planning and tracking between two frames under the constraints of acceleration, and interpolate the flight instructions between two frames (100 milliseconds, the drone's velocity command is 10 milliseconds / time).

[0082] In this embodiment, by specifying at least one of the position, velocity, acceleration, etc. of the starting point and the end point, Hermite interpolation can generate a path that better meets specific needs. This flexibility makes it very useful in many applications, such as robot paths. A smooth path is planned in a 2D plane space (actually a constrained plane in a 3D space, but due to acceleration and smoothness constraints, the actual path is in the vicinity of a perfect plane), and a PID controller is used to track this path. Specifically, the Hermite spline interpolation method is first used to generate a path according to the given starting and end point conditions, and then the current position, velocity and acceleration are continuously adjusted through the feedback mechanism of the PID controller to ensure that the system can follow this path. The Hermite spline path planning generates an ideal path, but in the actual motion process, it may be subject to various external interferences (such as sensor noise, environmental changes, etc.), resulting in deviations between the actual motion and the ideal path. The PID controller can adjust the system state in real time to reduce these deviations.

[0083] S160, looping through steps S110-S150 until the tower head insulator is identified by using the acquired real-time environment image.

[0084] In some embodiments, a target tracking algorithm is used to track each transmission line mask instance, and the state of the corresponding transmission line is updated according to the result of the tracking process. Specifically, the following steps can be used to implement the tracking process:

[0085] 1) determining the center point of each existing trajectory according to the existing trajectory set; and determining the line segment endpoints corresponding to each transmission line according to each transmission line mask instance.

[0086] 2) constructing a cost matrix based on the center points of the existing trajectories and the line segment endpoints corresponding to each transmission line; the cost function of the cost matrix is: the distance from a point to a straight line, where the point is the center point of each existing trajectory and the straight line is the straight line formed by the line segment endpoints. The straight line here can also be a straight line instance constructed by the transmission line mask instance.

[0087] 3) Using a multi-stage cascade matching algorithm, based on the cost matrix, determine the existing trajectory matching each transmission line.

[0088] Use the Hungarian algorithm or other linear assignment algorithm to find the best match between each existing trajectory and the transmission line in the cost matrix (cost function: point-to-line distance, point is the center point of the existing trajectory, line is the line where the line segment of the current frame is located).

[0089] 4) Update the status of the existing trajectory that is not matched to the transmission line in the trajectory status set to Lost; for the transmission line that is not matched to the existing trajectory, create a trajectory according to the transmission line mask instance, store the created trajectory in the existing trajectory set, set the status of the created trajectory to New, and update it into the trajectory status set; for the successfully matched transmission line and existing trajectory, update the position and center point of the corresponding existing trajectory according to the transmission line mask instance, and update the status of the existing trajectory in the trajectory status set to Activated.

[0090] 5) For an existing trajectory whose state is lost in the trajectory state set, if the duration of the existing trajectory being in the lost state exceeds a preset duration, the existing trajectory is removed from the existing trajectory set, and the state of the existing trajectory is removed from the trajectory state set or set to a removed state Removed.

[0091] In this embodiment, a matching list is returned, including which existing trajectories successfully match the transmission line, and which existing trajectories and transmission lines fail to match. For the existing trajectories and transmission lines that are successfully matched, the status of the corresponding existing trajectories is updated to "Activated", and its position, center point and other information are updated.

[0092] For existing trajectories that are not matched, their status needs to be set to "Lost" and their unupdated time step needs to be incremented to determine whether the maximum loss time has been exceeded. For unmatched transmission lines, they can be considered as new targets, and new trajectories can be created and added to the existing trajectory tracking list and marked as "New".

[0093] When an existing trajectory fails to match any transmission line in consecutive time steps, and after a certain number of time steps (no update time steps), the set maximum loss time (max_time_lost) is exceeded, the existing trajectory will be considered invalid and set to "Removed". This is because after such a long time without update, it indicates that the target may have disappeared, in order to avoid unnecessary calculation and resource consumption.

[0094] In the cascade matching process, the state information of the existing trajectory is used to predict its possible position in the next frame and match it with the observation data of the current frame (i.e. the power line in the current frame). By continuously optimizing this matching process, the tracker can track the power line more accurately and cope with various complex situations.

[0095] In the cascade matching process, we first try to match the existing track in the "lost" state with the new observation data in the current frame (i.e., the power line in the current frame). If the match is successful, the target state is updated to "activated" or "tracked" and normal tracking continues. If the match fails, the existing track is kept in the "lost" state and the matching attempt is continued in the next frame.

[0096] In some embodiments, predicting the position and speed of the drone in the next frame can be achieved using the following steps:

[0097] 1) The sub-models of each motion mode in the interactive multi-model are used to process the transmission line mask instances respectively to obtain the predicted position and predicted speed corresponding to each motion mode.

[0098] The motion modes include a motion mode corresponding to stillness, a motion mode corresponding to uniform linear motion, and a motion mode corresponding to uniform accelerated linear motion.

[0099] 2) Perform weighted summation on the predicted positions corresponding to each motion mode to obtain the position of the drone in the next frame.

[0100] 3) Take the weighted sum of the predicted speeds corresponding to each motion mode to obtain the speed of the drone in the next frame.

[0101] The use of the interactive multiple model (IMM) method usually involves the need to deal with complex and dynamically changing systems. In the process of visual side line simulation, it involves switching between multiple motion modes: hovering, constant speed, acceleration and deceleration, etc. In addition, when UAVs are operating at high altitudes in the field, they are easily affected by wind and vibration interference. Conventional Kalman cannot solve the stable tracking problems of complex motion modes such as high maneuverability, interference, and seamless switching of motion modes.

[0102] IMM is a method for modeling the motion of an exciting target. Its main idea is adaptive weighted averaging. The specific algorithm implementation varies according to specific needs. This embodiment combines the modeling of three motion modes: stillness, uniform linear motion, and uniformly accelerated linear motion. Each of these three motion modes is modeled and implemented using linear Kalman. Of course, IMM has other applications and implementation methods. This embodiment selects these three motion modes based on the trade-offs made from observations of real data: the more models there are, the more robust the tracking is, but the slower the running speed is. In addition, this embodiment can dynamically adjust the weight and selection of the model according to the changes in the target motion mode to maintain the accuracy and robustness of tracking.

[0103] The present disclosure abstracts the transmission line into the center point of a line segment. The transmission line or existing trajectory state & observation includes: state (position, velocity, acceleration of the midpoint): x, y, dx, dy, ax, ay.

[0104] Observation (the position of the midpoint, obtained by the current frame wire extraction algorithm mentioned above): x, y.

[0105] Kalman filter is an efficient recursive filter suitable for estimating the state of a system from a series of incomplete and noisy measurements. It is widely used in control engineering, navigation systems, signal processing and computer vision, especially for target tracking.

[0106] Working principle of Kalman filter:

[0107] The Kalman filter is based on two main assumptions:

[0108] Linear Models: The system can be described as linear difference or differential equations.

[0109] Gaussian noise: Both process noise and observation noise are zero-mean Gaussian white noise.

[0110] Its workflow includes two steps:

[0111] Predict: Predict the current state based on the previous state.

[0112] Update: Use the current measurement value to correct the predicted state.

[0113] Specifically, the Kalman filter is calculated using the following formula:

[0114] Prediction stage:

[0115] Prediction state: x'(k) = A*x(k-1) + B*u(k)

[0116] Prediction error covariance: P'(k) = A*P(k-1)*A^T+Q

[0117] Update phase:

[0118] Kalman gain: K(k) = P'(k)*H^T*(H*P'(k)*H^T+R)^(-1)

[0119] Update status: x(k) = x'(k) + K(k)*(z(k)-H*x'(k))

[0120] Update error covariance: P(k) = (IK(k)*H)*P'(k)

[0121] in,

[0122] x(k) is the state vector, which may represent position, velocity, etc. in target tracking;

[0123] u(k) is the control input vector (if there is external control);

[0124] A and B are the state transfer matrix and control input matrix respectively;

[0125] P(k) is the error covariance matrix, which is used to measure the uncertainty of the estimate;

[0126] Q is the process noise covariance matrix;

[0127] z(k) is the measurement vector;

[0128] H is the observation matrix, which maps the state space to the measurement space;

[0129] R is the measurement noise covariance matrix;

[0130] K(k) is the Kalman gain, which determines how much new information should be adopted.

[0131] Application of Kalman filter in target tracking: In the target tracking task of computer vision, Kalman filter is usually used to predict the position of the object in the future frame and update the state based on the actual detection results. Specific application scenarios include but are not limited to:

[0132] Single target tracking: Even if the target is temporarily lost, the Kalman filter can provide reasonable predictions and help recover the target.

[0133] Multiple Target Tracking: Multiple objects can be tracked simultaneously by maintaining an independent Kalman filter for each target.

[0134] Fusing sensor data: For example, combining camera and radar data to improve tracking accuracy.

[0135] The advantages of the Kalman filter are that it has a solid mathematical foundation, can effectively handle noise, and is computationally efficient. However, it assumes linearity and Gaussian distribution of the system, which may be limiting in some nonlinear or non-Gaussian environments. For such cases, the Extended Kalman Filter (EKF), Unscented Kalman Filter (UKF), or other nonlinear filtering methods may be more appropriate.

[0136] The target tracking algorithm process in this disclosure is as follows:

[0137] 1. Initialize the tracker (i.e., the execution subject in this embodiment)

[0138] -Select or define initialization parameters (such as time interval, maximum loss time, etc.)

[0139] - Initialize the tracker instance according to the parameters

[0140] 2. Input data preparation

[0141] - Get the image data of the current frame

[0142] -Extract line segments or target information from images (such as line segment endpoints, center points, etc.)

[0143] 3. Tracker initialization (if not initialized yet)

[0144] - Initialize the tracker with the extracted data

[0145] - Assign a unique ID to each target and initialize the state

[0146] 4. Target Tracking

[0147] -For each target, use a tracking algorithm (such as the IMM algorithm) to make predictions and updates

[0148] -Update the target state (such as position, speed, etc.) according to the data of the current frame

[0149] 5. Matching and updating

[0150] -Match the current frame's target with the target in the tracker

[0151] -Updated the status of the successfully matched target to "Active"

[0152] - For targets that are not matched successfully, increment their unupdated time steps

[0153] - If a time step has not been updated for more than the maximum missing time, update its state to "removed"

[0154] 6. Dealing with new goals

[0155] - For new targets that appear in the current frame but are not in the tracker

[0156] - Initializes a tracker instance for the new target and adds it to the tracking list

[0157] 7. Update tracker status

[0158] - Update the overall status of the tracker based on the matching and update results

[0159] - Remove targets marked as "remove"

[0160] 8. Get tracking results

[0161] - Extract the tracking results of the current frame from the tracker

[0162] -Includes target ID, location, status and other information

[0163] 9. Next frame processing

[0164] - If there is another frame of data, return to step 2 to continue processing

[0165] - Otherwise, end the tracking process

[0166] The above-mentioned target is a transmission line or a trajectory corresponding to a transmission line.

[0167] In the present disclosure, Hermite interpolation is applied to drone path planning and tracking. Because the drone sends a command every 10ms (frequency is 100Hz): speed signal (this speed signal is obtained by Hermite interpolation), and the tracking algorithm runs at a frequency (or frame rate) of 100ms each time, that is, 10HZ. There are 10 command gaps between two perceptions and trackings (that is, between two frames), and these 10 gaps are obtained by Hermite interpolation. The drone of the present disclosure is constrained in a plane, so it is a Hermite interpolation in 2D space.

[0168] How Hermite interpolation works: Assume there are two points A and B, and your speed at A and B is known. Hermite interpolation does the following: Find the right curve: It looks for a curve that passes through A and B and reflects your speed at A and B. Ensure a smooth transition: Make sure the change from A to B is smooth, not a sudden jump or turn.

[0169] The above-mentioned embodiments of the present disclosure solve the problem of difficulty in locking and following the line of the autonomous line simulation of the side of the drone vision, and are widely applicable to a variety of scenarios: large height difference and small height difference transmission towers; suitable for the side visual autonomous line simulation of a single transmission line and multiple parallel transmission lines; suitable for a variety of drone models: DJI Mavic 3E, M300, M350, etc. At the same time, it can be applied to low-computing platforms, such as Nvidia Tx2, and the transmission line can be perceived and extracted in real time and stably. In addition, the drone achieves safe, stable, precise and real-time control during the process of autonomous line simulation of the side vision.

[0170] Based on the same inventive concept, the present disclosure provides a purely visual side autonomous line-tracking device for a drone, the steps performed by the components of the device are the same or similar to those of the above method, so similar parts will not be repeated. Figure 4 As shown, a purely visual side autonomous line-tracking device for a drone of this embodiment includes:

[0171] The image acquisition module 410 is used to acquire real-time environment images captured by the camera carried by the drone.

[0172] The segmentation module 420 is used to perform semantic segmentation on the real-time environment image to obtain a power line mask; perform instance segmentation on the power line mask, and construct a power line mask instance according to the instance segmentation result.

[0173] The target tracking module 430 is used to track each transmission line mask instance separately using the target tracking algorithm, and update the state of the corresponding transmission line according to the result of the tracking process, and predict the position and speed of the drone in the next frame; wherein the target tracking algorithm includes a closed-loop finite state machine, an interactive multi-model, and a multi-level cascade matching algorithm.

[0174] The drone control module 440 is used to obtain the plane constraints of the drone flight and adjust the position and speed of the drone in the next frame according to the plane constraints.

[0175] The drone control module 440 is also used to plan the drone flight path by using the position of the drone in the current frame, the speed of the drone in the current frame, the position of the drone in the next frame, and the speed of the drone in the next frame, and to generate multiple flight instructions for the drone between the current frame and the next frame; the flight instructions include speed instructions and path instructions.

[0176] The loop control module 450 is used to control the image acquisition module, the segmentation module, the target tracking module, and the drone control module to loop through corresponding steps until the tower head insulator is identified using the acquired real-time environment image.

[0177] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a computer-readable storage medium.

[0178] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0179] like Figure 5 As shown, the device 500 includes a computing unit 510, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 520 or a computer program loaded from a storage unit 580 into a random access memory (RAM) 530. In the RAM 530, various programs and data required for the operation of the device 500 can also be stored. The computing unit 510, the ROM 520, and the RAM 530 are connected to each other via a bus 540. An input / output (I / O) interface 550 is also connected to the bus 540.

[0180] A number of components in the device 500 are connected to the I / O interface 550, including: an input unit 560, such as a keyboard, a mouse, etc.; an output unit 570, such as various types of displays, speakers, etc.; a storage unit 580, such as a disk, an optical disk, etc.; and a communication unit 590, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 590 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0181] The computing unit 510 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 510 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 510 performs the various methods and processes described above. For example, in some embodiments, any of the above methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 580. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM520 and / or communication unit 590. When the computer program is loaded into RAM530 and executed by the computing unit 510, one or more steps of any of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 510 may be configured to perform any of the methods described above in any other appropriate manner (e.g., by means of firmware).

[0182] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0183] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0184] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0185] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0186] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0187] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0188] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0189] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A purely visual side autonomous line-imitation method for unmanned aerial vehicles, characterized in that: include: S110, obtaining a real-time environment image collected by a camera carried by the UAV; S120, performing semantic segmentation on the real-time environment image to obtain a transmission line mask; Performing instance segmentation on the transmission line mask, and constructing a transmission line mask instance according to the instance segmentation result; S130, using a target tracking algorithm to track each transmission line mask instance respectively, and updating the state of the corresponding transmission line according to the result of the tracking process, and predicting the position and speed of the drone in the next frame; wherein the target tracking algorithm includes a closed-loop finite state machine, an interactive multi-model, and a multi-level cascade matching algorithm; S140, obtaining the plane constraint of the UAV flight, and adjusting the position and speed of the UAV in the next frame according to the plane constraint; S150, using the position of the drone in the current frame, the speed of the drone in the current frame, the position of the drone in the next frame, and the speed of the drone in the next frame, to plan the flight path of the drone, and generate multiple flight instructions for the drone between the current frame and the next frame; the flight instructions include speed instructions and path instructions; S160, looping through steps S110-S150 until the tower head insulator is identified by using the acquired real-time environment image.

2. The method according to claim 1, characterized in that The method of using the target tracking algorithm to track each transmission line mask instance respectively and updating the state of the corresponding transmission line according to the result of the tracking process includes: Acquire a closed-loop finite state machine; wherein the closed-loop finite state machine includes a current frame trajectory set, an existing trajectory set, and a trajectory state set for storing the state of the existing trajectory; Determine the center point of each existing trajectory according to the existing trajectory set; and determine the line segment endpoints corresponding to each transmission line according to each transmission line mask instance; A cost matrix is ​​constructed according to the center points of the existing trajectories and the line segment endpoints corresponding to each transmission line; the cost function of the cost matrix is: the distance from a point to a straight line, wherein the point is the center point of each existing trajectory and the straight line is the straight line formed by the line segment endpoints; Determine existing trajectories matching each transmission line based on the cost matrix using a multi-stage cascade matching algorithm; The status of the existing trajectory that is not matched to the transmission line in the trajectory state set is updated to lost; for the transmission line that is not matched to the existing trajectory, a trajectory is created according to the transmission line mask instance, and the created trajectory is stored in the existing trajectory set, and the status of the created trajectory is set to new, and updated into the trajectory state set; for the transmission line and the existing trajectory that are successfully matched, the position and center point of the corresponding existing trajectory are updated according to the transmission line mask instance, and the status of the existing trajectory in the trajectory state set is updated to active.

3. The method according to claim 2, characterized in that Also includes: For an existing trajectory whose state is lost in the trajectory state set, if the duration of the existing trajectory being in the lost state exceeds a preset duration, the existing trajectory is removed from the existing trajectory set, and the state of the existing trajectory is removed from the trajectory state set.

4. The method according to claim 1, characterized in that: The prediction of the position and speed of the drone in the next frame includes: The sub-models of each motion mode in the interactive multi-model are used to process the transmission line mask instances respectively, and the predicted position and predicted speed corresponding to each motion mode are obtained; The predicted positions corresponding to each motion mode are weighted summed to obtain the position of the drone in the next frame; The predicted speeds corresponding to each motion mode are weighted summed to obtain the speed of the drone in the next frame; The motion modes include a motion mode corresponding to stillness, a motion mode corresponding to uniform linear motion, and a motion mode corresponding to uniformly accelerated linear motion.

5. The method according to claim 1, characterized in that The two-dimensional plane corresponding to the plane constraint is parallel to the plane where the transmission line is located.

6. The method according to claim 1, characterized in that The method uses the position of the drone in the current frame, the speed of the drone in the current frame, the position of the drone in the next frame, and the speed of the drone in the next frame to plan the flight path of the drone, and generates multiple flight instructions for the drone between the current frame and the next frame, including: The Hermite spline interpolation method is used to generate a flight path based on the current frame drone position, the current frame drone speed, the next frame drone position, and the next frame drone speed; Through the feedback mechanism of the PID controller, when the UAV flies along the flight path, a path instruction for adjusting the position of the UAV and a speed instruction for adjusting the speed and acceleration are generated.

7. The method according to claim 1, characterized in that The performing instance segmentation on the transmission line mask comprises: The transmission line mask is instance segmented by searching for connected domains.

8. A purely visual side autonomous line-tracking device for unmanned aerial vehicles, characterized in that: include: An image acquisition module is used to acquire real-time environmental images collected by a camera carried on the drone; A segmentation module, used for performing semantic segmentation on the real-time environment image to obtain a power transmission line mask; Performing instance segmentation on the transmission line mask, and constructing a transmission line mask instance according to the instance segmentation result; A target tracking module is used to track each transmission line mask instance using a target tracking algorithm, and update the state of the corresponding transmission line according to the result of the tracking process, and predict the position and speed of the drone in the next frame; wherein the target tracking algorithm includes a closed-loop finite state machine, an interactive multi-model, and a multi-level cascade matching algorithm; The UAV control module is used to obtain the plane constraints of the UAV flight and adjust the position and speed of the UAV in the next frame according to the plane constraints; The drone control module is further used to plan the drone flight path by using the position of the drone in the current frame, the speed of the drone in the current frame, the position of the drone in the next frame, and the speed of the drone in the next frame, and to generate multiple flight instructions for the drone between the current frame and the next frame; the flight instructions include speed instructions and path instructions; The loop control module is used to control the image acquisition module, the segmentation module, the target tracking module, and the UAV control module to loop through corresponding steps until the tower head insulator is obtained by using the acquired real-time environmental image recognition.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images