Tethered UAV visual positioning method, system, electronic device and storage medium

Abstract

The embodiment of the present application provides a tethered drone visual positioning method, system, electronic device and storage medium, which belongs to the field of drone technology. The solution uses an infrared camera to obtain a positioning image set of a tethered drone in airspace, identifies and processes the positioning image set according to the thermal radiation characteristics of the tethered drone, obtains positioning information, uses an optical flow algorithm to perform benchmark comparison processing on the positioning information, obtains the displacement change of the tethered drone, and determines whether the tethered drone is offset according to the displacement change. When the tethered drone is offset, a closed-loop control algorithm is used to determine a control signal according to the displacement change, and the tethered drone is subjected to attitude control processing according to the control signal to control the tethered drone back to the reference position. The positioning can be achieved without relying on navigation technology, thereby improving the positioning accuracy of the tethered drone.

Description

Technical Field

[0001] The present application relates to the field of drone technology, and in particular to a tethered drone visual positioning method, system, electronic device and storage medium. Background Art

[0002] As an important branch of the drone field, tethered drones are characterized by combining drones with tethered cables, using ground power supply, and continuously transmitting power to the drone through the tethered cables, so that the drone can hover in the air for a long time and complete the specified work tasks.

[0003] At present, drones mainly rely on the Global Positioning System (GPS), and use the camera under the drone to obtain ground images and perform auxiliary positioning according to the algorithm. However, due to the long-term operation of tethered drones, GPS failure or deviation is prone to occur. In addition, although the flight range of tethered drones is not large, the actual situation on the ground is still unknown before the drone is applied. Since it is impossible to judge the actual application scenario in advance, the algorithm needs to do a lot of redundant training, resulting in a high difficulty in the algorithm and difficulty in achieving high-precision positioning. At the same time, the interference of the environment on the information captured by the camera in bad weather will also reduce the reliability of the image, resulting in a decrease in positioning accuracy. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a tethered drone visual positioning method, system, electronic device and storage medium, aiming to achieve positioning without relying on navigation technology and improve the positioning accuracy of the tethered drone.

[0005] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application proposes a tethered drone visual positioning method, the method comprising:

[0006] Acquire a positioning image set of the tethered UAV in the airspace through an infrared camera, wherein the infrared camera is deployed on the ground end, and the positioning image set includes a reference image and a comparison image;

[0007] Performing recognition processing on the positioning image set according to the thermal radiation characteristics of the tethered drone to obtain positioning information, wherein the positioning information includes a first position coordinate of the tethered drone in the reference image and a second position coordinate of the tethered drone in the comparison image;

[0008] Using an optical flow algorithm to perform benchmark comparison processing on the positioning information to obtain a displacement change of the tethered drone;

[0009] Determining whether the tethered drone is offset according to the displacement change;

[0010] When the tethered drone deviates, a closed-loop control algorithm is used to determine a control signal according to the displacement change;

[0011] The tethered drone is subjected to attitude control processing according to the control signal, and the tethered drone is controlled to return to a reference position.

[0012] In some embodiments, the step of acquiring a positioning image set of the tethered drone in the airspace by using an infrared camera includes the following steps:

[0013] Controlling the tethered drone to move to a preset position, wherein the preset position is set within the shooting range of the infrared camera;

[0014] When the tethered drone moves to a preset position, the infrared camera is controlled to capture a thermal infrared grayscale image to obtain a reference image;

[0015] Controlling the infrared camera to capture thermal infrared grayscale images at a preset frequency to obtain contrast images at different times;

[0016] A positioning image set is formed according to the reference image and the comparison image.

[0017] In some embodiments, the identifying and processing the positioning image set according to the thermal radiation characteristics of the tethered drone to obtain positioning information includes the following steps:

[0018] Determining a grayscale value threshold according to the thermal radiation characteristics of the tethered drone;

[0019] Determine the area in which the pixel grayscale value of each image in the positioning image set is greater than the grayscale value threshold as the target area;

[0020] Identify the shape of the tethered drone in the target area by an edge detection algorithm, and obtain a first position coordinate of the tethered drone in the reference image and a second position coordinate of the tethered drone in the comparison image;

[0021] Positioning information is obtained according to the first position coordinates and the second position coordinates.

[0022] In some embodiments, the use of an optical flow algorithm to perform a benchmark comparison process on the positioning information to obtain a displacement change of the tethered drone includes the following steps:

[0023] Using an optical flow algorithm, determine the pixel displacement of the tethered drone in a pixel coordinate system according to the first position coordinate and the second position coordinate;

[0024] Based on the mapping relationship between the pixel coordinate system and the world coordinate system, the displacement change of the tethered drone in the world coordinate system is determined according to the pixel displacement.

[0025] In some embodiments, the method of using an optical flow algorithm to determine the pixel displacement of the tethered drone in a pixel coordinate system according to the first position coordinates and the second position coordinates includes the following steps:

[0026] Obtaining optical flow information according to the first position coordinates and the second position coordinates;

[0027] The pixel displacement of the tethered drone in a pixel coordinate system is determined according to the optical flow information.

[0028] In some embodiments, the use of a closed-loop control algorithm to determine a control signal according to the displacement change comprises the following steps:

[0029] Performing proportional processing on the displacement change to obtain a proportional value of the displacement change;

[0030] Integrating the displacement change to obtain an integral value of the displacement change;

[0031] Performing differentiation processing on the displacement change to obtain a differential value of the displacement change;

[0032] Control signals for adjusting the yaw angle, the pitch angle and the roll angle are determined according to the proportional value, the integral value and the differential value.

[0033] In some embodiments, the tethered drone visual positioning method further includes the following steps:

[0034] The control signal is sent from the ground terminal to the tethered drone via a tethered cable.

[0035] To achieve the above object, another aspect of the embodiment of the present application provides a tethered drone visual positioning system, the system comprising:

[0036] The first module is used to obtain a positioning image set of the tethered UAV in the airspace through an infrared camera, wherein the infrared camera is deployed on the ground end, and the positioning image set includes a reference image and a comparison image;

[0037] A second module is used to identify and process the positioning image set according to the thermal radiation characteristics of the tethered drone to obtain positioning information, wherein the positioning information includes the first position coordinates of the tethered drone in the reference image and the second position coordinates in the comparison image;

[0038] The third module is used to perform a benchmark comparison process on the positioning information using an optical flow algorithm to obtain a displacement change of the tethered drone;

[0039] A fourth module is used to determine whether the tethered drone is offset according to the displacement change;

[0040] A fifth module is used for adopting a closed-loop control algorithm to determine a control signal according to the displacement change when the tethered drone deviates;

[0041] The sixth module is used to perform attitude control processing on the tethered drone according to the control signal, and control the tethered drone to return to a reference position.

[0042] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0043] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0044] The embodiments of the present application include at least the following beneficial effects: The present application provides a tethered drone visual positioning method, system, electronic device and storage medium. The solution uses an infrared camera to obtain a positioning image set of a tethered drone in airspace, which can be unaffected by light and can capture the movement of the drone in harsh environments. At the same time, there are few interference factors in the airspace, which can reduce the difficulty of the algorithm. The positioning image set is identified and processed according to the thermal radiation characteristics of the tethered drone to obtain positioning information, and the positioning information is subjected to benchmark comparison processing using an optical flow algorithm to obtain the displacement change of the tethered drone. The displacement change determines whether the tethered drone is offset. When the tethered drone is offset, a closed-loop control algorithm is used to determine a control signal according to the displacement change, and the tethered drone is subjected to attitude control processing according to the control signal to control the tethered drone back to the reference position. The positioning can be achieved based on offline image processing technology without relying on navigation technology, thereby improving the positioning accuracy of the tethered drone. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a tethered drone visual positioning method provided in an embodiment of the present application;

[0046] Figure 2 It is a schematic diagram of the application architecture system of the tethered drone visual positioning method provided in the embodiment of the present application;

[0047] Figure 3 It is a thermal infrared grayscale image taken at night provided by an embodiment of the present application;

[0048] Figure 4It is a thermal infrared grayscale image taken during the day provided in an embodiment of the present application;

[0049] Figure 5 is a thermal infrared grayscale image of a tethered drone that has deviated, as provided in an embodiment of the present application;

[0050] Figure 6 It is a thermal infrared grayscale image of the tethered drone provided in the embodiment of the present application after positioning and calibration using the method of the present application;

[0051] Figure 7 is a flow chart of a tethered drone visual positioning method provided by another embodiment of the present application;

[0052] Figure 8 It is a structural schematic diagram of a tethered drone visual positioning system provided in an embodiment of the present application;

[0053] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0055] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0056] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] Before describing the embodiments of the present application in detail, the technical background or technical evolution context on which the embodiments of the present application are based is first introduced.

[0059] A tethered drone is a type of drone that obtains energy by connecting to a ground power supply system via a cable. Compared to drones that obtain energy by carrying their own batteries or fuel, tethered drones can hover in the air for a long time without interruption. However, long-term hovering also makes the probability of tethered drone GPS failure and deviation greater than that of conventional drones.

[0060] At present, most drones use fixed cameras under the aircraft to assist in navigation and positioning. However, there is huge uncertainty and complexity in processing data using ground images as input. It is necessary to process a large amount of complex and changeable ground environment information such as terrain undulations and vegetation coverage. These factors make algorithm design difficult and it is difficult to ensure that the ideal positioning accuracy can be achieved in all scenarios. In addition, due to the uncertainty of actual scene applications, the algorithm needs to undergo a large amount of redundant training to cover as many environmental conditions as possible, which increases the complexity of drone development and maintenance. In order to solve the problem of algorithm complexity, it is often necessary to equip high-performance processors and large-capacity storage devices to improve the drone's data processing capabilities and algorithm operation efficiency to ensure computing power, resulting in high software and hardware costs.

[0061] In view of this, a method, system, electronic device and storage medium for visual positioning of a tethered drone are provided in the embodiments of the present application. The visual positioning method of a tethered drone provided in the embodiments of the present application relates to the field of drone technology. The visual positioning method of a tethered drone provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the visual positioning method of a tethered drone, etc., but is not limited to the above forms.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0063] Reference Figure 1 , Figure 1 is an optional flow chart of the visual positioning method for a tethered drone provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0064] Step S101, obtaining a positioning image set of the tethered UAV in airspace through an infrared camera, wherein the infrared camera is deployed on the ground, and the positioning image set includes a reference image and a comparison image.

[0065] Step S102: performing recognition processing on the positioning image set according to the thermal radiation characteristics of the tethered drone to obtain positioning information, wherein the positioning information includes the first position coordinates of the tethered drone in the reference image and the second position coordinates in the comparison image.

[0066] Step S103, using an optical flow algorithm to perform benchmark comparison processing on the positioning information to obtain the displacement change of the tethered drone.

[0067] Step S104, judging whether the tethered drone is offset according to the displacement change.

[0068] Step S105: When the tethered drone deviates, a closed-loop control algorithm is used to determine a control signal according to the displacement change.

[0069] Step S106, performing attitude control processing on the tethered drone according to the control signal, and controlling the tethered drone to return to the reference position.

[0070] Considering that there are usually relatively clean backgrounds in airspace, such as blue sky and clouds, these backgrounds will cause less interference in the image. In airspace, the contrast between the tethered drone as a target and the surrounding environment is high, which is more conducive to algorithm processing than the complex image information on the ground. Under the same computing power, processing airspace images is faster and more accurate than processing ground images. At the same time, considering the interference of bad weather such as rain and dark environment on the information captured by the camera, in this embodiment, reference is made to Figure 2 The infrared camera is deployed on the ground. Specifically, the infrared camera can capture the infrared radiation emitted by the object for imaging. It is not restricted by light conditions and has the characteristics of all-weather imaging. It replaces the camera using optical imaging so that the infrared image of the tethered drone can still be captured in low-light environments or in harsh conditions such as fog and rain, thereby improving the reliability and stability of the image.

[0071] In order to intuitively feel that it is feasible and distinguishable to obtain the heat of the tethered drone and the airspace background, for example, this embodiment uses a 640*512 resolution infrared camera to shoot a 24-inch drone hovering at an altitude of 80m during the day and at night. Figure 3 This is the picture taken at night. Figure 4 The images are taken during the day. According to the images taken during the day and at night, the infrared camera is not limited by light conditions and can image all day long. In addition, when the tethered drone overlaps with the clouds, the infrared camera can capture the temperature difference between the tethered drone and the background environment based on the principle of thermal imaging, and visually distinguish and identify the tethered drone. It is understood that the overlap between the tethered drone and the clouds includes the tethered drone being above the clouds or the tethered drone being below the clouds. For example, according to Figure 4 It can be seen that although the clouds will form a certain shielding effect, the infrared camera can still clearly distinguish the tethered drone by using the temperature difference between the clouds and the tethered drone.

[0072] Next, the infrared camera is used to obtain a set of positioning images of the tethered UAV in the airspace. Specifically, the infrared camera can take pictures of the tethered UAV in the airspace at a fixed time or according to preset conditions. The baseline image refers to the picture taken when the tethered UAV arrives at the preset position; the comparison image uses the current picture taken in real time as a comparison with the baseline image to detect the position change of the tethered UAV.

[0073] After obtaining the positioning image set, the positioning image set is identified and processed according to the thermal radiation characteristics of the tethered drone to obtain positioning information. Specifically, the temperature of the drone body is significantly higher than the temperature of the airspace environment, and a preliminary screening is performed in the infrared image according to the temperature to obtain the approximate area of ​​the drone in the image. Then, a more accurate drone shape and positioning information based on the infrared image are obtained through an edge detection algorithm, where the positioning information includes the first position coordinates of the tethered drone in the reference image and the second position coordinates in the comparison image.

[0074] Furthermore, the optical flow algorithm is used to perform benchmark comparison processing on the positioning information to obtain the displacement change of the tethered drone. The first position coordinate in the benchmark image is compared with the second position coordinate in the comparison image, and the displacement vector of the tethered drone between the benchmark image and the comparison image is calculated using the optical flow algorithm, thereby obtaining the displacement change of the tethered drone in the world coordinate system.

[0075] Whether the tethered drone has deviated is determined based on the displacement change. Specifically, the calculated displacement change is compared with a preset displacement threshold. When the displacement change is greater than or equal to the displacement threshold, it can be considered that the tethered drone has deviated; when the displacement change is less than the displacement threshold, it can be considered that the tethered drone has not deviated.

[0076] When the tethered drone is detected to be offset, a closed-loop control algorithm is used to determine the control signal according to the displacement change. The closed-loop control algorithm is a control method based on the feedback principle. The error between the expected target value and the actual measured value is used as input. The differential control of the input is realized by increasing the feedback of the output, thereby realizing automatic adjustment of the output so that the output reaches the expected target. Common control strategies of closed-loop control algorithms include proportional (P) control, integral (I) control and differential (D) control. Among them, proportional control is to adjust the output proportionally according to the size of the error to eliminate the error and reduce oscillation; integral control is to perform an integral operation on the error, which can accumulate past errors and eliminate steady-state errors; differential control is to perform a differential operation on the error, which can respond to the rate of change, predict future trends, and reduce oscillation. Proportional integral (PI) control, proportional differential (PD) control and proportional integral differential (PID) control can be performed through a combination strategy. In this embodiment, the displacement change obtained by the optical flow algorithm is used as the input of the closed-loop control algorithm, and the control coefficient is adjusted by the PID algorithm according to the displacement change, and the corresponding control signal is output. The tethered UAV is subjected to attitude control processing according to the control signal, and the tethered UAV is controlled to return to the reference position (i.e. the preset position).

[0077] For example, Figure 3The position of the tethered drone is the reference position. During the operation of the tethered drone, no manual flight command is received. After a period of time, the infrared camera captures the infrared image at the current moment. Figure 5 As shown, Figure 3 As a reference image, Figure 5 As a comparison image, a positioning image set is formed. The positioning image set is identified and processed according to the thermal radiation characteristics of the tethered drone to obtain positioning information. The positioning information is subjected to benchmark comparison processing using the optical flow algorithm to obtain the displacement change of the tethered drone. Offset detection is performed based on the displacement change to determine that the tethered drone has shifted to the right. The control signal is determined by processing the displacement change using the PID algorithm. The yaw angle, pitch angle, and roll angle of the tethered drone are adjusted according to the control signal to perform attitude control and control the movement of the tethered drone. The controlled image is obtained through an infrared camera, and the current image is identified to determine whether the position of the tethered drone after control is still offset from the reference position. If there is still an offset, the control signal is re-determined through the PID algorithm based on the currently calculated displacement change until the tethered drone returns to the reference position, referring to Figure 6 The method provided in this embodiment can be used to control the drone to fly to Figure 3 The reference position in .

[0078] In some embodiments, step S101 may include but is not limited to steps S201 to S204.

[0079] Step S201, controlling the tethered drone to move to a preset position, wherein the preset position is set within the shooting range of the infrared camera.

[0080] Step S202, when the tethered drone moves to a preset position, the infrared camera is controlled to capture a thermal infrared grayscale image to obtain a reference image.

[0081] Step S203, controlling the infrared camera to capture the thermal infrared grayscale image at a preset frequency to obtain comparison images at different times.

[0082] Step S204: forming a positioning image set according to the reference image and the comparison image.

[0083] In this embodiment, the infrared camera is deployed on the ground side. Figure 2, an infrared camera is installed on the main body of the tethered power supply end at the ground end, and the lens of the infrared camera is set upward to obtain the thermal infrared grayscale image of the tethered drone in the airspace. It should be noted that the infrared camera is fixedly connected to the main body of the tethered power supply end so that the shooting range of the infrared camera (i.e., the airspace) remains unchanged. After fixing the infrared camera, the tethered drone is controlled to move to a preset position, wherein the preset position is set within the shooting range of the infrared camera. When the tethered drone moves to the preset position, the infrared camera is controlled to capture the thermal infrared grayscale image to obtain a reference image. Exemplarily, after the tethered drone executes the flight command to fly to the preset position, if the tethered drone does not receive a flight command within 5 seconds, it is considered to be in a hovering state, triggering the infrared camera at the ground end to store a snapshot to obtain a reference image.

[0084] Then, the infrared camera is controlled to capture the thermal infrared grayscale image at a preset frequency to obtain comparison images at different times. For example, during the hovering operation of the tethered drone, the infrared camera continuously captures real-time images, and the thermal infrared grayscale image captured by the tethered drone at the current moment is determined as the comparison image according to the timestamp, and the position of the tethered drone in the airspace is monitored through the comparison image. A positioning image set is formed based on the reference image and the comparison image for subsequent data processing.

[0085] In some embodiments, step S102 may include but is not limited to steps S301 to S304.

[0086] Step S301, determining a gray value threshold according to the thermal radiation characteristics of the tethered drone.

[0087] Step S302: determine the area in each image in the positioning image set where the pixel grayscale value is greater than the grayscale value threshold as the target area.

[0088] Step S303, identifying the shape of the tethered drone in the target area by using an edge detection algorithm, and obtaining the first position coordinates of the tethered drone in the reference image and the second position coordinates of the tethered drone in the comparison image.

[0089] Step S304, obtaining positioning information according to the first position coordinates and the second position coordinates.

[0090] In this embodiment, due to the mechanical structure and working characteristics of the tethered drone, its motor, battery, electronic equipment, etc. will generate specific heat when it is running, and then radiate infrared light outward, making the temperature of the tethered drone higher than the surrounding environment. In the thermal infrared grayscale image, a grayscale value of 0 is presented as black, and a grayscale value of 255 is presented as white. Therefore, the higher the grayscale value, the brighter the color. Different grayscales represent different temperatures. The change in grayscale value in the image directly reflects the temperature distribution and change on the surface of the object. The thermal radiation characteristics of the tethered drone are characterized by brighter areas, while the environment in the airspace such as the blue sky and clouds usually appears as darker areas. Therefore, the grayscale value threshold can be determined based on the thermal radiation characteristics of the tethered drone.

[0091] Next, the grayscale threshold is used to effectively distinguish the drone from the background area in the thermal infrared grayscale image. Specifically, the images in the positioning image set are traversed to determine the pixel grayscale value of each pixel point. The area where the pixel grayscale value of each image in the positioning image set is greater than the grayscale threshold is determined as the target area. The target area is screened out by the grayscale threshold, and the potential area containing the tethered drone can be preliminarily screened out, paving the way for more accurate positioning and identification of the tethered drone in the future.

[0092] After determining the target area, the shape of the tethered UAV is identified in the target area by an edge detection algorithm, and the first position coordinates of the tethered UAV in the reference image and the second position coordinates in the comparison image are obtained.

[0093] Optionally, a Canny edge detection algorithm is used, and a Gaussian filter is applied to smooth the target area to eliminate noise, and the gradient strength and direction of each pixel in the target area are calculated. The target area is subjected to non-maximum suppression processing, and the gradient strength of each pixel in the gradient direction is compared with that of adjacent pixels. If the gradient strength of the current pixel is not the largest, it is suppressed to filter non-edge points, thereby making the blurred boundary clear.

[0094] Furthermore, dual threshold detection is performed by presetting a first threshold and a second threshold of the gradient strength, wherein the first threshold is greater than the second threshold. Pixels whose gradient strength is greater than the first threshold can be considered as strong edge points, pixels whose gradient strength is less than the second threshold can be considered as non-edge points, and pixels whose gradient strength is less than the first threshold and greater than the second threshold can be considered as potential edge points. After dual threshold detection, the 8-neighborhood of the potential edge point is detected. When the potential edge point is connected to the strong edge point, the potential edge point is a true edge point, otherwise, it is regarded as a non-edge point and suppressed. Finally, through the edge connection operation, the isolated edge points or edge fragments are connected to form a complete edge contour, and the shape of the tethered drone is identified.

[0095] After obtaining the shape of the drone, a pixel coordinate system can be established in the image. For example, a pixel coordinate system is established with the pixel in the upper left corner of the image as the coordinate origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis. According to the shape of the drone identified by the edge recognition algorithm, the first position coordinates of the tethered drone are determined in the reference image, and the second position coordinates of the tethered drone are determined in the comparison image. According to the first position coordinates and the second position coordinates, the positioning information of the tethered drone in the positioning image set is obtained.

[0096] In some embodiments, step S103 may include but is not limited to steps S401 to S402.

[0097] Step S401, using an optical flow algorithm, determining a pixel displacement of the tethered drone in a pixel coordinate system according to the first position coordinate and the second position coordinate.

[0098] Step S402: Based on the mapping relationship between the pixel coordinate system and the world coordinate system, the displacement change of the tethered drone in the world coordinate system is determined according to the pixel displacement.

[0099] In this embodiment, an optical flow algorithm is used to calculate the change and correlation of pixels in the reference image and the comparison image in the time domain, find the corresponding relationship between the reference image and the comparison image, and thus calculate the motion of the tethered drone.

[0100] Optical flow algorithms include dense optical flow algorithms and sparse optical flow algorithms. Dense optical flow algorithms are an image registration method that matches images point by point and requires calculating the offset of all pixels in the image. Sparse optical flow algorithms are the opposite of dense optical flow algorithms. Sparse optical flow algorithms do not calculate each pixel of the image point by point, but specify a set of key points for tracking. Compared with dense optical flow algorithms, sparse optical flow algorithms only select some key points in the image for calculation, which greatly reduces the amount of calculation. Under the same hardware conditions, images can be processed faster, reducing the requirements for computing power, thereby simplifying the difficulty of the algorithm. At the same time, in scenarios where tethered drones need to run for a long time, since sparse optical flow algorithms do not require high computing power and require fewer resources, the overall cost of the solution can be reduced.

[0101] Therefore, this embodiment adopts a sparse optical flow algorithm, takes the shape of the tethered drone as a feature point, and determines the pixel displacement of the tethered drone in the pixel coordinate system according to the first position coordinate and the second position coordinate. After obtaining the pixel displacement, it is also necessary to convert the pixel displacement of the tethered drone in the pixel coordinate system into the actual displacement change in the world coordinate system. Specifically, the calibration parameters of the infrared camera are obtained, wherein the calibration parameters include an internal parameter matrix (such as focal length, principal point coordinates, etc.) and an external parameter matrix (such as rotation matrix and translation vector, etc.). The points in the world coordinate system are converted to the camera coordinate system using the external parameter matrix, and then the points in the camera coordinate system are projected to the image coordinate system through the internal parameter matrix and the perspective projection principle. Finally, the image coordinate system and the pixel coordinate system are on the same plane. The points in the image coordinate system are converted to the pixel coordinate system through simple translation and scaling, thereby establishing a mapping relationship between the pixel coordinate system and the world coordinate system, and the displacement change of the tethered drone in the world coordinate system is determined according to the pixel displacement based on the mapping relationship between the pixel coordinate system and the world coordinate system.

[0102] In some embodiments, step S105 may include but is not limited to steps S501 to S502.

[0103] Step S501, obtaining optical flow information according to the first position coordinates and the second position coordinates.

[0104] Step S502: determining the pixel displacement of the tethered drone in the pixel coordinate system according to the optical flow information.

[0105] In this embodiment, optical flow information is obtained according to the first position coordinate and the second position coordinate, and the optical flow information includes the gradient I of the tethered drone in the x direction. x (i.e. the rate of change in the x-direction), the gradient I in the y-direction y (i.e., the rate of change in the y-axis direction) and the gradient I in the time direction t (i.e. the rate of change over time t). The optical flow equation is established based on the optical flow information, and the expression of the optical flow equation is:

[0106] I x ·u+I y ·v+I t =0(1)

[0107] Among them, u and v are the movement speed of the pixel in the x and y directions respectively.

[0108] Assuming spatial consistency, the pixels in the same image have the same motion, and the optical flow velocity (u, v) is solved by the least squares method.

[0109] The optical flow velocity is multiplied by the time interval between two frames of images to obtain the pixel displacement of the tethered drone in the pixel coordinate system.

[0110] In some embodiments, step S106 may include but is not limited to steps S601 to S602.

[0111] Step S601, performing proportional processing on the displacement variation to obtain a proportional value of the displacement variation.

[0112] Step S602, integrating the displacement variation to obtain an integral value of the displacement variation.

[0113] Step S603, performing differentiation processing on the displacement variation to obtain a differential value of the displacement variation.

[0114] Step S604, determining control signals for adjusting the yaw angle, the pitch angle and the roll angle according to the proportional value, the integral value and the differential value.

[0115] In this embodiment, the closed-loop control algorithm is implemented by a PID controller, which includes three parts: proportional control, which is used to perform proportional processing on the displacement change to obtain the proportional value of the displacement change; integral control, which is used to perform integral processing on the displacement change to obtain the integral value of the displacement change; and differential control, which is used to perform differential processing on the displacement change to obtain the differential value of the displacement change.

[0116] The proportional value, integral value and differential value are weighted and summed to obtain the control signal for adjusting the yaw angle, pitch angle and roll angle. The output u(t) of the PID controller can be expressed by the following formula:

[0117]

[0118] Among them, e(t) is the current error representing the difference between the target value and the actual measured value (i.e., the displacement change), K p is the proportionality coefficient, K i is the integration coefficient, K d is the differential coefficient.

[0119] In some embodiments, the tethered drone visual positioning method may also include but is not limited to step S701.

[0120] Step S701, sending a control signal from the ground terminal to the tethered drone via a tethered cable.

[0121] In this embodiment, the power carrier technology is applied to send the control signal from the ground end to the tethered drone through the tethered cable. Figure 2 ,The ground end exchanges data with the tethered UAV through the tethered cable, and sends the control signal calibrated by PID algorithm to the UAV end to control the UAV to perform actions.

[0122] For example, the tethered cable serves as a bridge connecting the tethered drone and the ground end. When it is used as a power line to provide a stable power supply for the tethered drone, it can also be used as a transmission medium to realize data transmission, communication and control functions on the power line.

[0123] Specifically, the tethered cable contains a positive power supply line and a negative power supply line, and both the ground end and the tethered drone are equipped with power carrier modules, wherein the power carrier module on the ground end is a communication modulation device, and the power carrier module on the tethered drone is a communication demodulation device. The ground end modulates the control signal to be transmitted through the communication modulation device, superimposes the modulated signal on the positive and negative power supply lines and transmits it through the tethered cable, and demodulates it through the communication demodulation device on the tethered drone to restore it to the control signal of the original information.

[0124] It is understandable that the length of the tethering cable can be selected according to the specific scenario and working requirements of the tethered drone while meeting the quality of power carrier communication, and the embodiments of the present application do not impose specific limitations.

[0125] The following is a detailed introduction and description of the solution of the embodiment of the present invention in conjunction with a specific application example. Figure 7 After determining the working location of the tethered UAV and fixing the tethered ground end, send the flight command to control the tethered UAV to take off. When the tethered UAV flies to the preset position according to the flight command and does not receive the flight command within 5s, it is considered to be in a hovering state, triggering an infrared camera on the tethered ground end facing upward to take an infrared image and obtain a reference image.

[0126] After the ground end obtains the thermal infrared reference information of the airspace and the UAV based on the reference image, the infrared image is obtained in real time while the tethered UAV is hovering. The infrared image at the current moment is used as a comparison image, and the infrared image is processed through an object recognition algorithm. The self-heating of the tethered UAV is significantly higher than that of the surrounding environment, and the temperature is preliminarily screened to obtain the approximate position of the tethered UAV. At the same time, a more accurate UAV shape and the x, y coordinates based on the infrared image at this time are obtained through an edge detection algorithm.

[0127] The coordinates in the comparison image and the coordinates in the reference image are compared to determine whether the current position is offset from the reference position. If the result is negative, no processing is required.

[0128] If the result is yes, the PID algorithm is used to process the offset information to obtain the calibration information for controlling the UAV flight. The ground end sends the calibration information to the tethered UAV through the power carrier to drive the UAV to perform flight actions, and feeds back the results of the algorithm processing to obtain the adjusted infrared image, and re-compare the reference position to determine whether there is still an offset. If so, the algorithm optimization is repeated until the tethered UAV flies back to the reference position.

[0129] The algorithms used in this embodiment can all be calculated in an offline environment, are not affected by the network, and can achieve the positioning and offset correction of the tethered drone without relying on GPS and RTK, and control the tethered drone to hover at the designated work location to complete the specified work task. At the same time, deploying the algorithm processing on the ground side can optimize the design cost of the drone and transfer the structure, hardware, software, and algorithm costs required for auxiliary positioning to the ground side.

[0130] Reference Figure 8 The embodiment of the present application also provides a tethered drone visual positioning system, which can implement the above-mentioned tethered drone visual positioning method, and the system includes:

[0131] The first module is used to obtain a positioning image set of the tethered UAV in the airspace through an infrared camera, wherein the infrared camera is deployed on the ground end, and the positioning image set includes a reference image and a comparison image;

[0132] The second module is used to identify and process the positioning image set according to the thermal radiation characteristics of the tethered drone to obtain positioning information, wherein the positioning information includes the first position coordinates of the tethered drone in the reference image and the second position coordinates in the comparison image;

[0133] The third module is used to perform benchmark comparison processing on the positioning information using the optical flow algorithm to obtain the displacement change of the tethered drone;

[0134] The fourth module is used to determine whether the tethered drone is offset according to the displacement change;

[0135] The fifth module is used to use a closed-loop control algorithm to determine the control signal according to the displacement change when the tethered drone deviates;

[0136] The sixth module is used to perform attitude control on the tethered UAV according to the control signal, and control the tethered UAV to return to the reference position.

[0137] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned tethered drone visual positioning method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0139] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] Reference Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0141] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0142] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the tethered drone visual positioning method of the embodiment of this application;

[0143] Input / output interface 903, used to implement information input and output;

[0144] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0145] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0146] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0147] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned tethered drone visual positioning method is implemented.

[0148] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0150] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0151] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0152] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0153] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0154] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0155] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A tethered drone visual positioning method, characterized in that: The method comprises the following steps: Acquire a positioning image set of the tethered UAV in the airspace through an infrared camera, wherein the infrared camera is deployed on the ground end, and the positioning image set includes a reference image and a comparison image; Performing recognition processing on the positioning image set according to the thermal radiation characteristics of the tethered drone to obtain positioning information, wherein the positioning information includes a first position coordinate of the tethered drone in the reference image and a second position coordinate of the tethered drone in the comparison image; Using an optical flow algorithm to perform benchmark comparison processing on the positioning information to obtain a displacement change of the tethered drone; Determining whether the tethered drone is offset according to the displacement change; When the tethered drone deviates, a closed-loop control algorithm is used to determine a control signal according to the displacement change; The tethered drone is subjected to attitude control processing according to the control signal, and the tethered drone is controlled to return to a reference position.

2. The method according to claim 1, characterized in that The method of obtaining a positioning image set of a tethered UAV in airspace by using an infrared camera comprises the following steps: Controlling the tethered drone to move to a preset position, wherein the preset position is set within the shooting range of the infrared camera; When the tethered drone moves to a preset position, the infrared camera is controlled to capture a thermal infrared grayscale image to obtain a reference image; Controlling the infrared camera to capture thermal infrared grayscale images at a preset frequency to obtain contrast images at different times; A positioning image set is formed according to the reference image and the comparison image.

3. The method according to claim 2, characterized in that The step of identifying and processing the positioning image set according to the thermal radiation characteristics of the tethered drone to obtain positioning information comprises the following steps: Determining a grayscale value threshold according to the thermal radiation characteristics of the tethered drone; Determine the area in which the pixel grayscale value of each image in the positioning image set is greater than the grayscale value threshold as the target area; Identify the shape of the tethered drone in the target area by an edge detection algorithm, and obtain a first position coordinate of the tethered drone in the reference image and a second position coordinate of the tethered drone in the comparison image; Positioning information is obtained according to the first position coordinates and the second position coordinates.

4. The method according to claim 1, characterized in that: The method of using an optical flow algorithm to perform a benchmark comparison process on the positioning information to obtain a displacement change of the tethered drone includes the following steps: Using an optical flow algorithm, determine the pixel displacement of the tethered drone in a pixel coordinate system according to the first position coordinate and the second position coordinate; Based on the mapping relationship between the pixel coordinate system and the world coordinate system, the displacement change of the tethered drone in the world coordinate system is determined according to the pixel displacement.

5. The method according to claim 4, characterized in that The method of using an optical flow algorithm to determine the pixel displacement of the tethered drone in a pixel coordinate system according to the first position coordinates and the second position coordinates includes the following steps: Obtaining optical flow information according to the first position coordinates and the second position coordinates; The pixel displacement of the tethered drone in a pixel coordinate system is determined according to the optical flow information.

6. The method according to claim 5, characterized in that The closed-loop control algorithm is used to determine the control signal according to the displacement change, including the following steps: Performing proportional processing on the displacement change to obtain a proportional value of the displacement change; Integrating the displacement change to obtain an integral value of the displacement change; Performing differentiation processing on the displacement change to obtain a differential value of the displacement change; Control signals for adjusting the yaw angle, the pitch angle and the roll angle are determined according to the proportional value, the integral value and the differential value.

7. The method according to claim 1, characterized in that The tethered drone visual positioning method also includes the following steps: The control signal is sent from the ground terminal to the tethered drone via a tethered cable.

8. A tethered drone visual positioning system, characterized in that: The system comprises: The first module is used to obtain a positioning image set of the tethered UAV in the airspace through an infrared camera, wherein the infrared camera is deployed on the ground end, and the positioning image set includes a reference image and a comparison image; A second module is used to identify and process the positioning image set according to the thermal radiation characteristics of the tethered drone to obtain positioning information, wherein the positioning information includes the first position coordinates of the tethered drone in the reference image and the second position coordinates in the comparison image; The third module is used to perform a benchmark comparison process on the positioning information using an optical flow algorithm to obtain a displacement change of the tethered drone; A fourth module is used to determine whether the tethered drone is offset according to the displacement change; A fifth module is used for adopting a closed-loop control algorithm to determine a control signal according to the displacement change when the tethered drone deviates; The sixth module is used to perform attitude control processing on the tethered drone according to the control signal, and control the tethered drone to return to a reference position.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images