Drone monitoring and control system

Abstract

To provide a drone monitoring control system capable of upgrading work efficiency and diminishing human resources.SOLUTION: A drone monitoring control system includes a drone, a moving vehicle designed to carry the drone, an arithmetic unit, and a display device. The drone incorporates a work payload and camera. The drone performs work payload output work. The arithmetic unit outputs a first environment image according to a first image taken by the camera, generates a flight trajectory of the drone and a moving trajectory of the moving vehicle according to a work dataset including a target position of a target object and 3D topographic data, and controls the drone so that the drone can move to a set position along the flight trajectory. When a distance between the target position and set position falls below a pre-set distance, the arithmetic unit controls the drone so that the drone will stay at the set position. The display device displays the first environment image and 3D topographic data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a drone surveillance and control system. [Background technology]

[0002] The presence of moisture, salt, and dust in the air makes it easy for dirt to adhere to the surfaces of various facilities, such as high-voltage transmission towers, wind turbine blades, and the glass of high-rise buildings. Summary of the Invention

[0003] Accordingly, the present disclosure provides a drone surveillance and control system.

[0004] According to an embodiment of the present disclosure, a drone monitoring and control system includes at least one drone, at least one mobile vehicle, a computing device, and a display device. The at least one drone is equipped with a work payload and at least one camera, the at least one camera configured to capture a first image, and the at least one drone is configured to perform an output operation of the work payload. The at least one mobile vehicle is configured to carry the at least one drone. The computing device is connected to the at least one drone, and is configured to output a first environmental image according to the first image captured by the at least one camera, and generate a flight trajectory of the at least one drone and a movement trajectory of the at least one mobile vehicle according to a work dataset corresponding to the output operation, where the work dataset includes a target position of a target object and three-dimensional terrain data of the target position. The computing device is further configured to control the at least one drone to move to a set position according to the flight trajectory, and to stay at the set position when a distance between the target position and the set position is less than a predetermined distance. A display device is coupled to the computing device, the display device being configured to display the first environment image and the three-dimensional terrain data.

[0005] From the above description, it can be seen that the drone monitoring and control system according to one or more embodiments of the present disclosure can improve work efficiency and reduce human resources. [Brief explanation of the drawings]

[0006] The present disclosure will be more fully understood from the detailed description and accompanying drawings set forth below. The detailed description and accompanying drawings are given by way of example only and are therefore not intended to limit the disclosure. [Figure 1] FIG. 1 is a block diagram illustrating a drone surveillance control system according to one embodiment of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram illustrating a drone surveillance and control system according to one embodiment of the present disclosure. [Figure 3] FIG. 1 is a block diagram illustrating a drone surveillance control system according to another embodiment of the present disclosure. [Figure 4] 4(a) to 4(c) are schematic diagrams showing sub-images corresponding to different fields of view, and FIG. 4(d) is a schematic diagram showing the first environmental image. [Figure 5] FIG. 10 is a block diagram illustrating a drone surveillance control system according to yet another embodiment of the present disclosure. [Figure 6] FIG. 10 is a schematic curve diagram for determining whether to control drone cooperation according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0007] In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. From the description, claims, and drawings disclosed in the specification, those skilled in the art can easily understand the concepts and features of the present invention. The following embodiments further illustrate various aspects of the present invention, but do not limit the scope of the present invention.

[0008] Please refer to Fig. 1, which is a block diagram illustrating a drone monitoring and control system according to an embodiment of the present disclosure. As shown in Fig. 1, the drone monitoring and control system 1 includes at least one drone 10, at least one mobile vehicle 11, a computing device 12, and a display device 13. The computing device 12 is connected to the drone 10 and the display device 13.

[0009] The drone 10 includes a work payload and at least one camera 101. The camera 101 is configured to capture a first image. The camera 101 may be an omnidirectional camera, and the omnidirectional image (first image) captured by the camera 101 may be used as the first environmental image. Alternatively, multiple cameras 101 may be present, and images (first images) captured by cameras in different directions may be stitched together to form the first environmental image. The drone 10 is configured to perform output operations of the work payload. When the drone monitoring and control system 1 is used for high-altitude cleaning operations, the work payload may include a liquid such as water or cleaning liquid, compressed air, or an electrical energy storage device. When the drone monitoring and control system 1 is used for rust removal on high-rise towers, the work payload may include a rust removal paint or an electrical energy storage device for laser rust removal. This disclosure does not limit the content of the work payload. The output operation may include spraying water, cleaning liquid, rust removal paint, or an air jet onto a target, irradiating a target with directional electromagnetic energy (such as a laser), sonic energy, etc. Targets include electricity transmission towers (tension-supported transmission towers, suspension transmission towers, etc.), wind turbines, high towers, skyscrapers, solar panels, etc. Further targets include insulators on electricity towers, wind turbine blades, glass on towers and skyscrapers, etc.

[0010] The mobile vehicle 11 may have a landing pad configured to support the drone 10. In other words, the mobile vehicle 11 may carry the drone 10 and deliver it to a set location, after which the drone 10 may perform an output operation of the work payload. The set location may be a location where the drone 10 performs the output operation, and may be located between a predetermined takeoff location of the drone 10 and a target location described below. The mobile vehicle 11 may include one or more of an automobile, a watercraft, an airship, a bicycle, and a motorcycle.

[0011] The computing device 12 is configured to output a first environmental image according to a first image captured by the camera 101, and the computing device 12 may stitch together multiple first images captured by the camera 101 into the first environmental image. The computing device 12 generates a flight trajectory of the drone 10 and a movement trajectory of the mobile vehicle 11 according to a working dataset corresponding to the output operation, where the working dataset includes a target position of the target and three-dimensional terrain data of the target position. The number of targets may be one or more, and the present disclosure is not limited thereto. The computing device 12 may execute at least one of a particle swarm optimization (PSO) algorithm, a genetic algorithm, and an ant colony optimization (ACO) algorithm on the working dataset to generate the flight trajectory and the movement trajectory.

[0012] For example, the target object is a power transmission tower, the output work is cleaning the power transmission tower, the target position includes the coordinates of the power transmission tower (e.g., latitude and longitude, etc.), the three-dimensional terrain data includes the altitude of the target position, the height of the target object and the heights of surrounding obstacles, etc., the flight trajectory includes the trajectory of the drone 10 flying from the above-mentioned predetermined takeoff position to the target position, the movement trajectory includes the trajectory of the moving vehicle 11 moving from the current position to the predetermined takeoff position, and the distance between the predetermined takeoff position and the target position is not greater than the first preset distance.

[0013] The computing device 12 is further configured to control the drone 10 to move to a set location according to a flight trajectory, and to control the drone 10 to stay at the set location if the distance between the target location and the set location is shorter than a second preset distance. The computing device 12 may be further configured to control the drone 10 to perform an output task after controlling the drone 10 to stay at the set location. In other words, when the drone 10 stays at the set location, the drone 10 may start performing the output task. If the camera 101 is a depth camera, the computing device 12 may use the camera 101 to calculate the distance between the drone 10 and the target object. If the camera 101 is not a depth camera, the computing device 12 may perform image recognition on a first image captured by the camera 101 to obtain an image of the target object and determine the distance between the target position and the set location based on the image of the target object and its reference size (actual size). For example, the computing device 12 may calculate a reduction ratio based on the size of the target object in the image and the reference size of the target object, and determine the distance between the target position and the set location based on the reduction ratio. The set position may be the destination of the flight trajectory, and the starting point of the flight trajectory may be the destination of the movement trajectory of the moving vehicle 11 (ie, the default take-off position).

[0014] If the height of the target is not less than the preset height, the first preset distance may be equal to or greater than the second preset distance, and the second preset distance may be less than the height of the target. If the height of the target is less than the preset height, the first preset distance may be equal to or less than the second preset distance, and the second preset distance may not be less than the height of the target. For example, if the target is a power transmission tower, the height of the target is not less than the preset height; if the target is a solar panel, the height of the target is less than the preset height. The preset height is 50 meters, the first preset distance is 40 meters, and the second preset distance is 2 to 3 meters. The height and preset distance values ​​described herein are merely examples, and the present disclosure is not limited thereto.

[0015] The computing device 12 may be located on the mobile vehicle 11, or may also be a cloud computing device. The computing device 12 may include one or more processors, such as a central processing unit, a graphics processing unit, a microcontroller, a programmable logic controller, or any other processor with signal processing capabilities.

[0016] The display device 13 receives the first environmental image and the three-dimensional terrain data output by the computing device 12 and displays the first environmental image and the three-dimensional terrain data. The three-dimensional terrain data displayed by the display device 13 may include one or more of letters, numbers, and terrain patterns. The computing device 12 may also connect to a server, such as a meteorological agency or an environmental observation station, to obtain the current wind speed, current wind direction, current temperature, and current light direction at the target location. The display device 13 may further be configured to display the distance between the target position and the set position, the current wind speed at the target position, the current wind direction at the target position, the current temperature at the target position, the current light direction at the target position, the remaining operational payload of the drone, and the remaining power of the drone. In one embodiment, the display device 13 may be at least one or more flat-panel monitors, curved monitors, projectors, head-mounted displays, or combinations thereof.

[0017] A drone monitoring and control system according to one or more embodiments of the present disclosure may improve work efficiency and reduce human resources.

[0018] In one embodiment, the working dataset may further include at least one of the type of target object, a plurality of candidate parking positions for the mobile vehicle 11, the available flight time of the drone 10, the current wind direction and current wind speed at the target position, and a default operation time of the target object. The computing device 12 may use one of the candidate parking positions closest to the target position as the default takeoff position. Furthermore, the working dataset may further include the position of an object of the same type as the target object in an operation range (e.g., a range centered on the target position and having a radius of a second preset distance), the type of the object (e.g., type of voltage, number of insulators, number of insulator strings and insulator hanging method, etc.), and an estimated operation duration of an output operation, etc. The computing device 12 may further generate at least one of an operation process of the drone 10 corresponding to the output operation, a shortest flight time of the drone 10, a default takeoff position and a default landing position of the drone 10, a timing for replenishing the operation payload, a charging time of the drone 10, and a number of charging cycles of the drone 10 according to the working dataset, and the computing device 12 may run at least one of a particle swarm optimization algorithm, a genetic algorithm, and an ant colony optimization algorithm on the working dataset to generate data such as the above-mentioned operation process.

[0019] Therefore, the drone monitoring and control system can plan output operations, including planning the drone's flight trajectory, number of flights, timing of replenishing the work payload, number of battery replacements, etc., based on the spatial information of the output operation, the target position, and the durability of the drone.

[0020] In one embodiment, the computing device 12 may control the camera 101 to capture a second image before controlling the drone 10 to perform the output task, and control the camera 101 to capture a third image after controlling the drone 10 to perform the output task, and compare the second image with the third image to determine the cleanliness level of the target. Specifically, the computing device 12 may compare multiple blocks in the second image with multiple corresponding blocks in the third image to determine the cleanliness level. Alternatively, the computing device 12 may learn to classify clean and non-clean target images through color feature values ​​and a scale-invariant feature transform (SIFT) feature value algorithm, and then determine the corresponding cleanliness level of the third image.

[0021] Please refer to Figures 1 and 2. Figure 2 is a schematic diagram illustrating a drone monitoring and control system according to one embodiment of the present disclosure. Figure 2 shows an exemplary schematic diagram corresponding to Figure 1. As shown in Figure 2, a mobile vehicle 11 may include a landing platform 111 configured to carry a drone 10, and a display device 13 is disposed inside the mobile vehicle 11. In the example of Figure 2, the computing device 12 is a cloud computing device, but the computing device 12 may also be disposed inside the mobile vehicle 11, and the present disclosure is not limited thereto.

[0022] Furthermore, the drone monitoring and control system may further include a supply chamber disposed inside the mobile vehicle 11. The supply chamber may include a working payload. Thus, when the drone 10 lands on the landing platform 111, the working payload of the drone 10 may be replenished.

[0023] Please refer to Figure 3, which is a block diagram showing a drone monitoring and control system according to another embodiment of the present disclosure. As shown in Figure 3, the drone monitoring and control system 2 includes at least one drone 20, at least one mobile vehicle 21, a computing device 22, and a display device 23. The computing device 22 is connected to the drone 20 and the display device 23. The mobile vehicle 21, the computing device 22, and the display device 23 may be the same as the mobile vehicle 11, the computing device 12, and the display device 13 shown in Figures 1 and 2, respectively, and details thereof will not be repeated here.

[0024] The drone 20 includes a first camera 201 and a second camera 202. The first camera 201 and the second camera 202 are connected to the computing device 22. The first camera 201 and the second camera 202 are respectively disposed at different positions on the drone 20. The first camera 201 and the second camera 202 are configured to acquire multiple initial images. The computing device 22 is further configured to generate a first environment image using the initial images having different fields of view. In other words, the first camera 201 may be configured to acquire a first initial image, and the second camera 202 may be configured to acquire a second initial image. The computing device 22 may run an image tracking algorithm on the first initial image and the second initial image to determine at least one of the target and the drone 20 in the first initial image and the second initial image, and stitch the first initial image and the second initial image into the first environment image.

[0025] It should be noted that the number of cameras shown in FIG. 3 is merely an example, and the number of cameras on a drone may be two or more, and the present disclosure is not limited thereto.

[0026] Please refer to FIGS. 4(a) to 4(d), where FIGS. 4(a) to 4(c) are schematic diagrams illustrating sub-images corresponding to different fields of view, and FIG. 4(d) is a schematic diagram illustrating a first environmental image. FIG. 4(a) illustrates a first sub-image corresponding to a first field of view when the camera is disposed within the drone 20. FIG. 4(b) illustrates a second sub-image corresponding to a second field of view when the camera is disposed outside another drone. FIG. 4(c) illustrates a third sub-image corresponding to a third field of view when the camera is disposed outside a moving vehicle. The first field of view may be a front view of the drone 20, the second field of view may be a top view of another drone, and the third field of view may be a front view of the moving vehicle. The first environmental image shown in FIG. 4(d) may be an image generated by a computing device by transforming and splicing multiple initial images. The computing device may perform an image tracking algorithm on the images captured by the camera so that each of the first to third sub-images represents at least one of the target A1 and the drone 20. Furthermore, as described above, the camera of the drone 20 may be a depth camera, and the computing device may determine depth information corresponding to the first sub-image through the camera, and further, the computing device may perform image recognition on the second sub-image and the third sub-image to determine the distance between the drone 20 and the target A1.

[0027] As shown in FIG. 4(d), the first environmental image displayed by the display device may include first sub-image IMG1 to fifth sub-image IMG5. For example, the first sub-image IMG1 may be the first sub-image shown in FIG. 4(a), the second sub-image IMG2 may be the second sub-image shown in FIG. 4(b), the third sub-image IMG3 may be the third sub-image shown in FIG. 4(c), the fourth sub-image IMG4 may be an image corresponding to a front view, and the fifth sub-image IMG5 may be an image corresponding to a rear view. Note that the examples of the first sub-image IMG1 to the fifth sub-image IMG5 are merely illustrative, and the present disclosure does not limit the field of view of each sub-image or the position of each sub-image in the first environmental image.

[0028] Furthermore, multiple of the first to fifth sub-images IMG1 to IMG5 may be images captured by multiple cameras in multiple directions, for example, the multiple directions may include front, rear, right, left, top, and bottom views of the camera.

[0029] Please refer to Figure 5, which is a block diagram showing a drone monitoring and control system according to yet another embodiment of the present disclosure. As shown in Figure 5, the drone monitoring and control system 3 includes a first drone 30, a first mobile vehicle 31, a second drone 32, a second mobile vehicle 33, a computing device 34, and a display device 35. The computing device 34 is connected to the first drone 30, the second drone 32, and the display device 35. The computing device 34 and the display device 35 may be the same as the computing device 12 and the display device 13 shown in Figures 1 and 2, respectively, and details thereof will not be repeated here.

[0030] The first mobile vehicle 31 is configured to carry the first drone 30, and the second mobile vehicle 33 is configured to carry the second drone 32. A camera 301 is disposed on the first drone 30, and a camera 321 is disposed on the second drone 32. The first drone 30 and the second drone 32 may be the same as the drone 10 shown in Figures 1 and 2, and the cameras 301 and 321 may be the same as the camera 101 shown in Figures 1 and 2.

[0031] The computing device 34 may be configured to control the second drone 32 to perform the output work of the first drone 30 when it determines that the work payload of the first drone 30 is lower than a preset payload.

[0032] Specifically, the computing device 34 can collect status information of all operating drones, including the remaining payload, remaining power, and flight trajectory, and re-plan the output task of each operation according to the collected information. For example, if the computing device 34 determines that the working payload of the first drone 30 is lower than the preset payload, the working payload of the second drone 32 is not lower than the preset payload and is sufficient to continue at least a part of the remaining output task of the first drone 30, the computing device 34 can control the first drone 30 to return to the first mobile vehicle 31, replenish the working payload of the first drone 30, and control the second drone 32 to take over the remaining output task of the first drone 30.

[0033] Furthermore, when the computing device 34 determines that the power of the first drone 30 of the first drone 30 and the second drone 32 is lower than the preset power, the computing device 34 may be configured to control the second drone 32 to perform the output task of the first drone 30. Similarly, when the computing device 34 determines that the power of the first drone 30 is lower than the preset power, if the computing device 34 determines that the power of the second drone 32 is not lower than the preset power and is sufficient to take over at least a portion of the remaining output task of the first drone 30, the computing device 34 may control the first drone 30 to return to the first mobile vehicle 31 to charge the first drone 30, and control the second drone 32 to take over the remaining output task of the first drone 30.

[0034] Please refer to Figures 5 and 6, where Figure 6 is a schematic curve diagram for determining whether to control drone cooperation according to an embodiment of the present disclosure. Figure 6 shows the operating state of the first drone 30. According to Figure 6, the computing device 34 can determine whether to control the second drone 32 to take over the remaining output work of the first drone 30 using the difference between the planned amount of work payload and power and the actual amount, as well as the degree of completion of the output work.

[0035] In FIG. 6, curve S0 represents the degree of completion of the output work, curve S1 represents the planned power usage, curve S1' represents the actual power usage, curve S2 represents the planned work payload usage, and curve S2' represents the actual work payload usage.

[0036] Taking time t1 as an example, if the difference between curve S1 and curve S1' is greater than a predetermined value and the calculation device 34 determines that the power of the first drone 30 is not sufficient to complete the remaining output work, the calculation device 34 can determine whether to control the second drone 32 to take over the remaining output work of the first drone 30 depending on the remaining power of the second drone 32.

[0037] Also, taking time t1 as an example, if the difference between curve S2 and curve S2' is greater than a predetermined value and the calculation device 34 determines that the work payload of the first drone 30 is not sufficient to complete the remaining output work, the calculation device 34 may determine whether to control the second drone 32 to take over the remaining output work of the first drone 30 depending on the remaining work payload of the second drone 32.

[0038] The number of mobile vehicles and drones shown in FIG. 5 is merely an example, and the number of mobile vehicles and drones may be two or more, and the number of drones may be different from the number of mobile vehicles, and the present disclosure is not limited thereto.

[0039] For example, the number of targets may be three, and all three target locations are far from the road, thereby increasing the drone's flight distance. Therefore, the computing device may calculate that one mobile vehicle needs to carry two drones to perform the output task. In another embodiment, the number of targets may be four, and all four target locations are close to the road, thereby decreasing the drone's flight distance. Therefore, the computing device may calculate that one mobile vehicle needs to carry three drones to perform the output task. Furthermore, in an example where multiple drones are deployed on one mobile vehicle, the drones' default takeoff positions may be the same or different from each other, and the drones' default landing positions may be the same or different from each other.

[0040]

[0013] For the above reasons, the drone monitoring and control system according to one or more embodiments of the present disclosure can improve work efficiency and reduce human resources. Furthermore, the drone monitoring and control system may plan output work, including planning the drone's flight trajectory, the number of flights, the timing of replenishing the work payload, the number of battery replacements, etc., based on data such as spatial information of the output work and drone durability.

Claims

1. at least one drone having a work payload and at least one camera disposed thereon, the at least one camera configured to capture a first image, and the at least one drone configured to perform an output task for the work payload; at least one mobile vehicle configured to carry the at least one drone; a computing device connected to the at least one drone, configured to output a first environment image according to a first image captured by the at least one camera, and generate a flight trajectory of the at least one drone and a movement trajectory of the at least one mobile vehicle according to a task dataset corresponding to the output task, the task dataset including a target position of an object and three-dimensional terrain data of the target position, the computing device being configured to control the at least one drone to move to a set position set according to the flight trajectory, and to control the at least one drone to stay at the set position when a distance between the target position and the set position is less than a preset distance; a display device coupled to the computing device, the display device configured to display the first environmental image and the three-dimensional terrain data; If the height of the target is not less than the preset height, the preset distance is less than the height of the target; If the height of the target is less than the preset height, the preset distance is not less than the height of the target; Drone monitoring and control system.

2. the at least one camera is a plurality of cameras each disposed at a different position on the at least one drone, the plurality of cameras each configured to acquire a plurality of initial images; The computing device is further configured to generate a plurality of sub-images corresponding to different fields of view based on the plurality of initial images, and combine the plurality of sub-images into the first environment image. The drone monitoring and control system according to claim 1 .

3. The computing device is further configured to perform image recognition on the first image captured by the at least one camera to obtain an image of a target, and determine a distance between the target position and the set position according to the image of the target and a reference size of the target. The drone monitoring and control system according to claim 1 .

4. the working dataset further includes at least one of a type of the target object, a plurality of potential parking locations for the at least one moving vehicle, an available flight time for the at least one drone, a current wind direction and a current wind speed at the target location, and a predetermined working time for the target object; The drone monitoring and control system according to claim 1 .

5. The computing device is further configured to generate, according to the operation dataset, at least one of a work process corresponding to an output work of the at least one drone, a minimum flight time of the at least one drone, a takeoff position and a landing position of the at least one drone, a replenishment timing of the work payload, a charging time of the at least one drone, and a number of charging cycles of the at least one drone. The drone monitoring and control system according to claim 4.

6. the computing device runs at least one of a particle swarm optimization algorithm, a genetic algorithm, and an ant colony optimization algorithm on the working dataset to generate the flight trajectory and the movement trajectory. The drone monitoring and control system according to claim 1 .

7. the computing device executes at least one of a particle swarm optimization algorithm, a genetic algorithm, and an ant colony optimization algorithm on the task dataset to generate at least one of a task process corresponding to an output task of the at least one drone, a minimum flight time of the at least one drone, a takeoff and landing location of the at least one drone, a replenishment timing of the task payload, a charging time of the at least one drone, and a number of charging cycles of the at least one drone; The drone monitoring and control system according to claim 5.

8. The computing device is further configured to control the at least one drone to perform the output task after controlling the at least one drone to remain at the set position. The drone monitoring and control system according to claim 1 .

9. The computing device is further configured to: control the at least one camera to capture a second image before controlling the at least one drone to perform the output task; control the at least one camera to capture a third image after controlling the at least one drone to perform the output task; and compare the second image with the third image to determine a cleanliness level of the target object. The drone monitoring and control system according to claim 8.

10. the at least one drone is a plurality of drones, the at least one mobile vehicle is a plurality of mobile vehicles, the plurality of mobile vehicles are configured to carry the plurality of drones respectively, and when the computing device determines that the task payload of a first drone of the plurality of drones is lower than a preset payload, the computing device is further configured to control a second drone of the plurality of drones to perform the output task of the first drone. The drone monitoring and control system according to claim 8.

11. the at least one drone is a plurality of drones, the at least one mobile vehicle is a plurality of mobile vehicles, the plurality of mobile vehicles are configured to carry the plurality of drones respectively, and when the computing device determines that a power of a first drone of the plurality of drones is lower than a preset power, the computing device is further configured to control a second drone of the plurality of drones to perform an output task of the first drone. The drone monitoring and control system according to claim 8.

12. The display device is further configured to display at least one of a distance between the target position and the set position, a current wind speed, a current wind direction, a current temperature, and a current light direction. The drone monitoring and control system according to claim 1 .

Images