System for planning flying route of unmanned aircraft, unmanned aircraft, and method for inspecting bridge of unmanned aircraft

Abstract

To provide a system for planning a flying route of an unmanned aircraft, an unmanned aircraft, and a method for inspecting a bridge.SOLUTION: The unmanned aircraft recognizes the calibration curve in an image for a target bridge. The unmanned aircraft controls the unmanned aircraft so that the unmanned aircraft flies along the calibration curve and performs a bridge inspection task on the basis of a shield part and a non-shield part. The calibration start point and the calibration end point of the calibration curve correspond to the latitude coordinate and longitude coordinate of an actual world, respectively.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a system for planning a flight path of an unmanned aerial vehicle, an unmanned aerial vehicle, and a bridge inspection method. More specifically, the present invention relates to a system for planning a flight path of an unmanned aerial vehicle capable of performing a bridge inspection task, an unmanned aerial vehicle, and a bridge inspection method. [Background technology]

[0002] Recently, with the rapid development of unmanned aerial vehicle (UAV) technology, a large number of applications related to unmanned aerial vehicles have been proposed one after another. Since unmanned aerial vehicles have the advantages of cost and speed, one of the important applications is to use unmanned aerial vehicles to perform object inspection tasks (e.g., safety inspection of bridge structures).

[0003] In the prior art, an unmanned aerial vehicle needs to self-stabilize and hover in the air based on its coordinates (e.g., Global Positioning System (GPS)), height (e.g., barometer) and heading (e.g., magnetometer) during operation. However, when the GPS signal is blocked by buildings (e.g., under a bridge) or in bad weather, the unmanned aerial vehicle may pose a flight safety problem.

[0004] In the prior art, there are many positioning assistance methods for environments where GPS is weak, such as visual localization and mapping methods using vision, lidar, etc. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the environmental map based on visual navigation is based on the position of the unmanned aerial vehicle itself, and cannot mark the data position (i.e., world coordinate position) during patrol in real time and accurately, so it cannot be used during bridge inspection.

[0006] In light of this, how to provide technology that can perform bridge inspection tasks is an urgent goal that the industry must address. [Means for solving the problem]

[0007] One object of the present invention is to provide a system for planning a flight path for an unmanned aerial vehicle equipped with an imaging device capable of capturing images above, so that the unmanned aerial vehicle flies while capturing images from below a bridge, in which the unmanned aerial vehicle takes an image of a target bridge above, and when it detects that the image contains obscured and unobscured parts corresponding to a global positioning system, it recognizes a calibration line corresponding to the target bridge in the image, flies along the calibration line to perform a bridge inspection task, and plans a flight path for the unmanned aerial vehicle in which the calibration start point and calibration end point of the calibration line each correspond to latitude and longitude coordinates in the real world.

[0008] In some embodiments, controlling the unmanned aerial vehicle to fly along the calibration curve includes calculating the included angle between the flight direction of the unmanned aerial vehicle and the calibration curve and the distance between the unmanned aerial vehicle and the calibration curve, and adjusting the flight direction by controlling the included angle between the flight direction and the calibration curve to be less than or equal to an angle preset value, and the included angle between the flight direction and the calibration curve to be less than or equal to an angle preset value, and the distance between the unmanned aerial vehicle and the calibration curve to be less than or equal to a distance preset value.

[0009] In some embodiments, the unmanned aerial vehicle records an actual flight path to create an environment map, and calculates a coordinate transformation matrix corresponding to the environment map and the real world based on the actual flight path and the calibration curve.

[0010] In some embodiments, the actual flight path corresponds to a virtual coordinate system and the calibration curve corresponds to a world coordinate system, and the coordinate transformation matrix is ​​used to transform the virtual coordinate system to the world coordinate system.

[0011] In some embodiments, the coordinate transformation matrix is ​​used to transform and integrate coordinate system orientation and coordinate system scale.

[0012] In some embodiments, the unmanned aerial vehicle flies based on a plurality of virtual inspection routes and the environmental map, records the actual flight route flown by the unmanned aerial vehicle, subsequently creates the environmental map, and positions the unmanned aerial vehicle based on the coordinate transformation matrix to calculate actual coordinate values ​​corresponding to each of a plurality of inspection images corresponding to the target bridge.

[0013] In some embodiments, the unmanned aerial vehicle transforms a plurality of real-world inspection routes into the virtual inspection route using a coordinate transformation inverse matrix generated based on the coordinate transformation matrix.

[0014] In some embodiments, the environment map is created using a plurality of point clouds corresponding to the images.

[0015] In some embodiments, the environment map records at least one or a combination of imaging parameters, feature maps, coordinates, and timestamps corresponding to each of a plurality of locations.

[0016] One object of the present invention is to provide a system that plans a flight path for an unmanned aerial vehicle equipped with an imaging device capable of capturing images above, so that the unmanned aerial vehicle flies while capturing images from below a bridge, wherein the unmanned aerial vehicle takes an image of a target bridge above, and when it detects that the image includes a bridge bottom portion and a sky portion, it recognizes a calibration line corresponding to the target bridge in the image, flies along the calibration line to perform a bridge inspection task, and plans a flight path for the unmanned aerial vehicle in which the calibration start point and calibration end point of the calibration line each correspond to latitude and longitude coordinates in the real world.

[0017] Another object of the present invention is to provide a bridge inspection method for an unmanned aerial vehicle, the unmanned aerial vehicle including an upward camera device and a processor, the upward camera device being used to capture an image corresponding to a target bridge based on a tilt angle, the bridge inspection method including the steps of: recognizing a calibration curve corresponding to the target bridge in the image; and controlling the unmanned aerial vehicle to fly along the calibration curve based on the image to perform a bridge inspection task, wherein a calibration start point and a calibration end point of the calibration curve each correspond to latitude and longitude coordinates in the real world. Effect of the Invention

[0018] The bridge inspection technique (including at least an apparatus and a method) according to the present invention creates an environment map by recognizing the calibration curve of the target bridge whose position information is known and the actual flight path of the unmanned aerial vehicle. Then, the bridge inspection technique according to the present invention calculates a coordinate transformation matrix corresponding to the environment map and the real world based on the actual flight path and the calibration curve. The bridge inspection technique according to the present invention can further control the unmanned aerial vehicle to fly along the multiple virtual inspection paths to inspect the target bridge based on multiple virtual inspection paths and the environment map, and generate multiple inspection images corresponding to the target bridge and actual coordinate values ​​in the real world corresponding to each of the multiple inspection images. The bridge inspection technique according to the present invention can automatically and efficiently perform bridge inspection tasks and solve known technical shortcomings.

[0019] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, detailed technical details and embodiments of the present invention will be described in conjunction with the drawings, so that those skilled in the art in the technical field to which the present invention pertains can understand the technical features of the protected invention. [Brief description of the drawings]

[0020] [Figure 1] 1 is a schematic diagram showing the architecture and environment of an unmanned aerial vehicle of a first embodiment; [Diagram 2] 4A to 4C are schematic diagrams showing patrol operations and images in the first embodiment. [Diagram 3] FIG. 2 is a schematic diagram illustrating the output of some embodiments. [Figure 4] FIG. 1 is a schematic diagram illustrating the relationships of some embodiments. [Diagram 5] 1 is a schematic diagram illustrating flight results of some embodiments. [Figure 6] 13 is a flowchart showing a part of a bridge inspection method according to a third embodiment. [Figure 7] 1 is an operational flowchart of some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, the unmanned aerial vehicle and method for bridge inspection according to the present invention will be interpreted by the embodiments. However, such embodiments are not used to limit that the present invention must be implemented in any environment, application, or aspect as described in such embodiments. Therefore, the description of the embodiments is only for explaining the present invention, not for limiting the scope of the present invention. In the following embodiments and drawings, elements not directly related to the present invention are omitted and not shown, and the dimensions of each element and the dimensional proportion between elements are merely examples and are not used to limit the scope of the present invention. It should be understood that.

[0022] First, in the application environment of the present disclosure, the purpose is that an unmanned aerial vehicle equipped with a camera(s) needs to inspect a target bridge. In the present disclosure, the unmanned aerial vehicle 1 can perform an inspection operation corresponding to the target bridge.

[0023] A first embodiment of the present invention is an unmanned aerial vehicle 1, the architecture schematic of which is shown in Fig. 1. The unmanned aerial vehicle 1 includes a photographing device 11, a ranging and obstacle avoidance device 12, a processing device 13, a flight and control device 14, a positioning device 15 and a communication device 16.

[0024] In some embodiments, the photographing device 11 includes an upward photographing device. In some embodiments, the photographing device 11 further includes an inspection photographing device and a rotation mechanism (e.g., a gimbal). The distance measuring and obstacle avoidance device 12 includes one or more distance measuring devices, such as an optical flow sensor, an optical LiDAR (Light Detection And Ranging), an infrared distance measuring device, an ultrasonic distance measuring device, etc.

[0025] In some embodiments, the flight and control systems 14 include powered systems (e.g., rotors or fixed wings), inertial measurement systems (e.g., accelerometers and gyros), and environmental sensing systems (e.g., barometers, thermometers, magnetometers, etc.). The positioning systems 15 include GPS systems. The communication systems 16 include antenna systems.

[0026] In some embodiments, the processing device 13 includes a processor and is electrically connected to the imager 11 (e.g., the processor is electrically connected to the upward imager). The processor may be a variety of processing units, a Central Processing Unit (CPU), a microprocessor, or other computing device known to those skilled in the art.

[0027] The upward-facing photographing device is a photographing device that can photograph by moving the photographing lens toward the upper side of the unmanned aerial vehicle 1, and therefore includes, but is not limited to, a photographing device fixed facing upward. Therefore, any photographing device that can photograph the upper side, for example, a photographing device that is attached to the nose or tail and has a rotation mechanism (for example, a gimbal), should fall within the scope of this disclosure.

[0028] In some embodiments, in the operational application of the present disclosure, the unmanned aerial vehicle 1 can transmit relevant image content generated by the unmanned aerial vehicle 1 to a ground-side computer 3 or related computing device via the network NW for processing. In some embodiments, the unmanned aerial vehicle 1 can receive a bridge inspection command from the user-operated device 5 to perform a bridge inspection task, for example, between the 13th and 14th piers of a bridge numbered A3. In this embodiment, the upward-facing photographing device is used to photograph an image corresponding to the target bridge based on a tilt angle (e.g., 70 degrees upward from the horizontal plane) (e.g., operation S701 in FIG. 7). In some other embodiments, the unmanned aerial vehicle 1 rotates the photographing device using a rotation mechanism to photograph an image corresponding to the target bridge at a tilt angle.

[0029] In some embodiments, the image includes a bridge bottom portion and a sky portion corresponding to the target bridge.

[0030] For ease of understanding, please refer to Fig. 1. It shows an image schematic diagram 100 captured by an upward-facing photographing device. In the present disclosure, the upward-facing photographing device of the unmanned aerial vehicle 1 can obtain the image shown in Fig. 1, including the target object (i.e., the target bridge TB) and the calibration curve CL formed by the target object and the surrounding area, when looking up and photographing. For example, the surrounding area may be the sky including floating clouds FC, or other areas with large color difference from the target object.

[0031] The operation of the present disclosure can be divided into a calibration stage and a tour stage (for example, the calibration stage CS and tour stage IS in FIG. 7). First, in the calibration stage CS, the unmanned aerial vehicle 1 flies a line segment (i.e., a calibration curve) of known length and route information, and calculates a transformation matrix between a virtual coordinate system (i.e., a drawing coordinate system) and a real world coordinate system (i.e., a world coordinate system) by analyzing the calibration curve features in the image, and outputs the environmental map as a base map during operation in the tour stage IS. Then, in the tour stage IS, the base map is adopted to perform flight navigation, and the captured world coordinate values ​​are output.

[0032] Below, in this disclosure, the specific operations by which the unmanned aerial vehicle 1 performs the calibration phase CS will be described.

[0033] In this embodiment, the processor recognizes a calibration curve corresponding to the target bridge in the image (e.g., operation S702 in FIG. 7). Then, the processor controls the unmanned aerial vehicle 1 to fly along the calibration curve based on the image including the bridge bottom portion and the sky portion (e.g., operation S703 in FIG. 7). In some embodiments, each of the calibration start point and the calibration end point of the calibration curve corresponds to one latitude and longitude coordinate in the real world.

[0034] For ease of understanding, please refer to Fig. 2, which illustrates a patrol operation schematic diagram 200. In the patrol operation schematic diagram 200, a patrol object (i.e., a target bridge TB) is pre-selected, and an unmanned aerial vehicle 1 is placed near the object, and the unmanned aerial vehicle 1 can be controlled, for example, via a remote control, a computer, a mobile phone, or a smart terminal.

[0035] In this example, a linear region of a known length on the target bridge TB is preselected as the calibration curve CL, and the processor recognizes the calibration curve CL corresponding to the target bridge in the image, and controls the unmanned aerial vehicle 1 to fly along the calibration curve (i.e., fly from the calibration start point SP to the calibration end point EP). Note that the world coordinates (e.g., real-world latitude and longitude coordinates) of the calibration start point SP and calibration end point EP of the calibration curve CL are known, and this world coordinate system means numerical values ​​that can be interpreted by humans or software, such as latitude and longitude, bridge distance number, or coordinate values ​​in map data.

[0036] In some embodiments, during the calibration phase CS, the unmanned aerial vehicle 1 can simultaneously perform patrol tasks, reducing the cost of repeated flights of the unmanned aerial vehicle 1.

[0037] In the present disclosure, the unmanned aerial vehicle 1 can perform visual positioning navigation using captured images. In some embodiments, the unmanned aerial vehicle 1 further includes an optical flow module that overlooks the ground, and the optical flow module captures the visible range of the ground to obtain sensing data, and based on this, can calculate a movement vector within a short period of time and an offset amount of the unmanned aerial vehicle 1 relative to the ground. Based on this optical flow positioning function, the unmanned aerial vehicle can hover in the air in an environment without GPS, and can fly slightly around the surroundings.

[0038] In some embodiments, to make the determined information more accurate, the percentage of the area where the calibration curve CL is located on the target bridge TB corresponding to the image is approximately three-quarters of the image location (i.e., the target bridge TB occupies three-quarters of the area and the surrounding area occupies one-quarter of the area).

[0039] In some embodiments, the processor can reduce errors due to measurement by controlling the unmanned aerial vehicle 1 to fly close to the calibration curve based on preset parameters. Specifically, the processor calculates the included angle between the flight direction of the unmanned aerial vehicle 1 and the calibration curve and the distance between the unmanned aerial vehicle 1 and the calibration curve. Then, based on the included angle and the distance, the processor adjusts the flight direction of the unmanned aerial vehicle 1 by controlling the included angle between the flight direction and the calibration curve to be equal to or less than an angle preset value and the distance between the unmanned aerial vehicle 1 and the calibration curve to be equal to or less than a distance preset value.

[0040] For example, if the processor determines that the distance between the unmanned aerial vehicle 1 and the calibration curve is greater than the distance preset value, the processor controls the unmanned aerial vehicle 1 to perform a translational motion. Also, if the processor determines that the included angle between the flight direction of the unmanned aerial vehicle 1 and the calibration curve is greater than the angle preset value, the processor controls the unmanned aerial vehicle 1 to perform a steering motion.

[0041] For ease of understanding, please continue to refer to Fig. 2. Fig. 2 shows a schematic diagram 201 of an image captured by the unmanned aerial vehicle 1, including an object area TA and a calibration line CL. First, a virtual reference line RL is defined in the vertical direction of the image, and the reference line RL is used to fit the calibration line CL on the image to ensure that the unmanned aerial vehicle 1 flies along the line (i.e., the closer the reference line RL on which the unmanned aerial vehicle 1 flies to the calibration line CL, the better).

[0042] In this example, a perpendicular line VL from the center point CP of the reference line RL to the calibration line CL is calculated, and the distance (unit: pixel) between the reference line RL and the calibration line CL can be obtained from the length of the perpendicular line VL. The included angle AG between the reference line RL and the calibration line CL can be obtained from the slope of the reference line RL and the calibration line CL. The flight direction FD and relative position of the unmanned aerial vehicle 1 are adjusted based on the distance and included angle AG to minimize the distance and included angle AG, ensuring that the unmanned aerial vehicle 1 can fly along the calibration line CL.

[0043] Furthermore, since the calibration curve CL is located in a location where GPS can still be received (i.e., the GPS signal is not completely blocked), the unmanned aerial vehicle 1 can fly to the calibration start point SP of the calibration curve CL using methods such as GPS positioning and image recognition, and can obtain the image schematic 201 shown in Figure 2 at any point in time.

[0044] Then, in this embodiment, the processor records the actual flight path of the unmanned aerial vehicle 1 to create a map of the environment (eg, operation S704 in FIG. 7).

[0045] In some embodiments, the present disclosure performs visual localization and environment mapping (SLAM) operations to generate the environment map, and the environment map consists of multiple point clouds (Point Clouds) corresponding to multiple locations.

[0046] For example, the processor can extract a set of image features (e.g., edge or corner points) from the input image, compare the image features captured by the upward camera at a nearby time, and minimize the reprojection error of the features. Through the SLAM operation, the images captured by the upward camera can be correspondingly calculated to have coordinate values ​​in a virtual coordinate system, which represent the position of the upward camera (or the unmanned aerial vehicle 1) in the virtual space.

[0047] For example, see FIG. 3. A schematic diagram of the output of Visual SLAM is shown, including a created feature map 300 and a coordinate sequence 302 of an upward-facing camera (or unmanned aerial vehicle 1). The feature map 300 is made up of a point cloud and can represent a three-dimensional aspect of the photographed object, and any one point of the coordinate sequence 302 has a corresponding time stamp and a coordinate value in a virtual coordinate system (i.e., a drawing coordinate system) and can be used to refer to the corresponding image capture screen. From the above, the coordinate values ​​of the calibration start point and calibration end point of the calibration curve in the virtual coordinate system can be known from the start point 304 and end point 306 of the camera coordinate sequence 302.

[0048] Furthermore, once the unmanned aerial vehicle 1 completes the calibration flight, the unmanned aerial vehicle 1 can store the SLAM computational model as a base map (i.e., an environment map), which includes at least camera parameters, feature maps, coordinate sequences and corresponding timestamps, and this base map is used for positioning the patrol phase IS.

[0049] In some embodiments, the environment map records at least one or a combination of imaging parameters, feature maps, coordinate sequences, and timestamps corresponding to each of the multiple locations.

[0050] Finally, in this embodiment, the processor calculates a coordinate transformation matrix corresponding to the environment map and the real world based on the actual flight path and the calibration curve CL (eg, operation S705 in FIG. 7).

[0051] In addition, the actual flight path corresponds to a virtual coordinate system and the calibration curve CL corresponds to a world coordinate system, and the coordinate transformation matrix is ​​used to transform the virtual coordinate system into the world coordinate system.

[0052] Specifically, the coordinate transformation matrix includes matching operations for coordinate system orientation and coordinate system scale.

[0053] For ease of understanding, please refer to Fig. 4. Fig. 4 is a schematic diagram of the relationship between a world coordinate system 400 and a virtual coordinate system 402, which includes a calibration start point SP and a calibration end point EP of the calibration curve CL, and the length of the perpendicular line VL can be calculated as the distance from the calibration line CL using the position where the unmanned aerial vehicle 1 is located and the calibration line CL. An angle AG can be calculated using the flight direction FD of the unmanned aerial vehicle 1 and the calibration line CL.

[0054] In this example, TIFF2025036249000002.tif613 represents the coordinates of the calibration start point SP of the calibration curve CL in the world coordinate system 400, and the coordinates of point P 404 in space in the world coordinate system 400 and the virtual coordinate system 402. TIFF2025036249000003.tif613 and TIFF2025036249000004.tif612 and between them, rotation scaling matrix A 2x2 (i.e., coordinate transformation matrix A).

[0055] For example, the processor may calculate the coordinate transformation matrix A using the following equation: TIFF2025036249000005.tif731

[0056] In some embodiments, the processor may calculate an inverse matrix by a coordinate transformation matrix A to transform coordinates / paths expressed in the world coordinate system into coordinates / paths expressed in the virtual coordinate system.

[0057] For example, the processor may calculate the coordinate transformation inverse matrix using the following equation: TIFF2025036249000006.tif67 can be calculated. TIFF2025036249000007.tif733

[0058] Since the coordinate values ​​of the calibration start point SP and the calibration end point EP of the calibration curve CL in the world coordinate system 400 and the virtual coordinate system 402 are known, the above coordinate transformation matrix A can be obtained via singular value decomposition.

[0059] The specific operations of the unmanned aerial vehicle 1 during the patrol phase IS in this disclosure will be described below.

[0060] Upon completion of the calibration phase CS, the environment map and coordinate transformation matrix A can be obtained, and the unmanned aerial vehicle 1 loads the environment map in the patrol phase IS (e.g., operation S706 in FIG. 7) and reuses the previously created SLAM map creation coordinates. Then, using a method similar to that of the calibration phase CS, the processor controls the unmanned aerial vehicle 1 to fly to the calibration start point SP of the calibration curve CL and continue map creation (e.g., perform a visual SLAM operation), and the unmanned aerial vehicle 1 can estimate, based on the environment map, the origin at which the calibration start point SP of the calibration curve CL is located in the virtual coordinate system (e.g., operation S707 in FIG. 7).

[0061] Furthermore, during actual patrol, the unmanned aerial vehicle 1 can perform predetermined route tasks, continuously perform map creation during its flight period, and obtain the coordinates of the upward-facing shooting device (or the unmanned aerial vehicle 1) in the virtual coordinate system. Through the above-mentioned conversion operation, the shooting data can be converted from the virtual coordinate system to the world coordinate system, and the data positioning program can be completed.

[0062] Specifically, the processor controls the unmanned aerial vehicle 1 to fly along the multiple virtual inspection paths based on the multiple virtual inspection paths and the environmental map. Then, the processor records the actual flight path flown by the unmanned aerial vehicle 1 to subsequently create the environmental map. Finally, the processor positions the unmanned aerial vehicle 1 based on the coordinate transformation matrix to generate multiple inspection images corresponding to the target bridge and actual coordinate values ​​in the real world corresponding to each of the multiple inspection images.

[0063] In addition, the inspection route includes a plurality of inspection points, and the plurality of inspection points in the present disclosure are positions where the object to be inspected (i.e., the target bridge) needs to be inspected. Taking bridge inspection as an example, the plurality of inspection points are important positions for inspecting bridge safety structures (i.e., the unmanned aerial vehicle needs to take images corresponding to the object to be inspected from the inspection points), and the plurality of important positions need to be translated into world coordinates in order to be accurately interpreted.

[0064] In addition, in the patrol task mode, the unmanned aerial vehicle 1 can convert a series of waypoints (e.g., inspection points) in the world coordinate system into the virtual coordinate system through the above-mentioned conversion operation. As shown in FIG. 4, the starting point of this task is the origin of the virtual coordinate system (i.e., the calibration starting point SP), and when the task waypoint P404 is given, a virtual route 406 can be constructed, and the relative position and flight direction FD of the unmanned aerial vehicle 1 can be adjusted to minimize the relative distance and relative angle between the flight direction FD of the unmanned aerial vehicle 1 and the virtual route 406 until the target waypoint P404 is reached (e.g., operation S708 in FIG. 7). By repeating the above program, the flight task of each waypoint can be completed in sequence, and during the flight period, a map can be continuously created and the coordinates of the upward-facing shooting device (or the unmanned aerial vehicle 1) in the virtual coordinate system can be obtained, and the shooting data can be converted from the virtual coordinate system to the world coordinate system through the above-mentioned conversion operation, and the data positioning program can be completed (e.g., operation S709 in FIG. 7).

[0065] In some embodiments, the processor generates, based on the coordinate transformation matrix A, a coordinate transformation inverse matrix corresponding to the coordinate transformation matrix A. Then, based on the coordinate transformation inverse matrix, the processor transforms a plurality of inspection paths corresponding to the real world to generate the plurality of virtual inspection paths.

[0066] In addition, Figure 5 shows a schematic diagram of the patrol task flight result adopting Figure 3 as the base model, in which four sets of waypoints 500, 502, 504, 506 are preset, and by flying in the above-mentioned route task mode, a series of unmanned aerial vehicles' flight paths 508 can be generated, and at the same time, SLAM can be performed to create an environment map 510 to assist persistent positioning navigation.

[0067] In some embodiments, the multiple inspection points (or inspection paths) corresponding to the inspected object may be generated by a processor based on multiple inspection rules (e.g., an inspection point must be set every 5 meters) or may be pre-set by an expert.

[0068] In addition, the inspection points in this disclosure are also referred to as waypoints, and the waypoints may include information such as longitude, latitude, altitude, the drone's nose direction, or camera angle (i.e., the angle at which the drone captures the image at the waypoint).

[0069] In this embodiment, the processor can perform patrol operations by visual navigation of the unmanned aerial vehicle 1 based on the coordinate transformation matrix and environmental map generated in the calibration stage CS when there is no GPS signal (i.e., under the target bridge).

[0070] As can be seen from the above description, the unmanned aerial vehicle 1 according to the present invention creates an environmental map by recognizing the calibration curve of the target bridge whose position information is known and the actual flight path of the unmanned aerial vehicle 1. Then, the unmanned aerial vehicle 1 according to the present invention calculates a coordinate transformation matrix corresponding to the environmental map and the real world based on the actual flight path and the calibration curve. Note that the unmanned aerial vehicle 1 according to the present invention can further control the unmanned aerial vehicle 1 to fly along the multiple virtual inspection paths to inspect the target bridge based on multiple virtual inspection paths and the environmental map, and generate multiple inspection images corresponding to the target bridge and real coordinate values ​​in the real world corresponding to each of the multiple inspection images. The unmanned aerial vehicle 1 according to the present invention can automatically and efficiently perform bridge inspection tasks and solve known technical shortcomings.

[0071] The specific operations of the second embodiment will be described in detail below. In short, in addition to the operations performed in the first embodiment, in the second embodiment, an image captured by the upward photographing device corresponding to a target bridge includes an obstructed portion (i.e., no GPS signal) and an unobstructed portion (i.e., with a GPS signal) corresponding to a Global Positioning System (GPS) (for example, operation S701 in FIG. 7).

[0072] For example, as shown in FIG. 1, the left half of the image schematic 100 is an area where GPS signals cannot be received (i.e., the area below the target bridge area is an obstructed portion of the GPS signal), and the right half of the image schematic 100 is an area where GPS signals can be received (i.e., the unobstructed portion of the GPS signal).

[0073] Since the unmanned aerial vehicle 1 is located on the boundary of an area where GPS signals can be received, the processor can control the unmanned aerial vehicle 1 to perform the operation of the calibration stage CS along the calibration curve (i.e., from the calibration start point to the calibration end point) based on the image including the bridge bottom portion and the sky portion by the operation of the first embodiment. The processor performs the operation of the patrol stage IS based on the coordinate transformation matrix and the environmental map generated in the calibration stage CS.

[0074] The third embodiment of the present invention is a bridge inspection method, the flow chart of which is shown in FIG. 6. The bridge inspection method 600 is applied to an unmanned aerial vehicle, for example, the unmanned aerial vehicle 1 described in the first embodiment. The unmanned aerial vehicle includes an upward-facing camera and a processor, for example, the upward-facing camera and processor described in the first embodiment. The upward-facing camera is used to capture an image corresponding to a target bridge based on a tilt angle (for example, operation S701 in FIG. 7). The bridge inspection method 600 calculates a coordinate transformation matrix corresponding to the environmental map and the real world through steps S601 to S603.

[0075] In step S601, the unmanned aerial vehicle recognizes a calibration curve corresponding to the target bridge in the image (eg, operation S702 in FIG. 7).

[0076] Then, in step S603, the unmanned aerial vehicle controls, based on the image, the unmanned aerial vehicle to fly along the calibration line to perform a bridge inspection task (e.g., operation S703 in FIG. 7), where a calibration start point and a calibration end point of the calibration line each correspond to a real-world latitude and longitude coordinate.

[0077] In some embodiments, the bridge inspection method 600 further includes the steps of: recording an actual flight path of one of the unmanned aerial vehicles to create an environment map (e.g., operation S704 in FIG. 7 ), and calculating a coordinate transformation matrix corresponding to the environment map and the real world based on the actual flight path and the calibration curve (e.g., operation S705 in FIG. 7 ).

[0078] In some embodiments, the bridge inspection method 600 includes controlling the unmanned aerial vehicle to fly along a plurality of virtual inspection paths based on a plurality of virtual inspection paths and the environmental map, recording the actual flight path of the unmanned aerial vehicle to subsequently generate the environmental map, and positioning the unmanned aerial vehicle based on the coordinate transformation matrix to generate a plurality of inspection images corresponding to the target bridge and real-world actual coordinate values ​​corresponding to each of the plurality of inspection images.

[0079] In some embodiments, the multiple virtual inspection routes are generated based on the steps of generating a coordinate transformation inverse matrix corresponding to the coordinate transformation matrix based on the coordinate transformation matrix, and transforming multiple inspection routes corresponding to the real world based on the coordinate transformation inverse matrix to generate the multiple virtual inspection routes.

[0080] In some embodiments, the step of controlling the unmanned aerial vehicle to fly along the multiple virtual inspection paths further includes calculating an included angle between a flight direction of the unmanned aerial vehicle and the calibration curve and a distance between the unmanned aerial vehicle and the calibration curve, and adjusting the flight direction of the unmanned aerial vehicle based on the included angle and the distance by controlling the included angle between the flight direction and the calibration curve to be less than or equal to an angle preset value and the distance between the unmanned aerial vehicle and the calibration curve to be less than or equal to a distance preset value.

[0081] In addition to the above steps, the third embodiment can perform all the operations and steps of the unmanned aerial vehicle 1 described in the first and second embodiments, have similar functions, and achieve similar technical effects. Those skilled in the art can directly know how the third embodiment performs these operations and steps based on the above first and second embodiments, have similar functions, and achieve similar technical effects, so the description will be omitted.

[0082] As described above, the bridge inspection technique (including at least the device and the method) according to the present invention creates an environmental map by recognizing the calibration curve of the target bridge whose position information is known and the actual flight path of the unmanned aerial vehicle. Then, the bridge inspection technique according to the present invention calculates a coordinate transformation matrix corresponding to the environmental map and the real world based on the actual flight path and the calibration curve. Note that the bridge inspection technique according to the present invention can further control the unmanned aerial vehicle to fly along the multiple virtual inspection paths to inspect the target bridge based on multiple virtual inspection paths and the environmental map, and generate multiple inspection images corresponding to the target bridge and actual coordinate values ​​in the real world corresponding to each of the multiple inspection images. The bridge inspection technique according to the present invention can automatically and efficiently perform bridge inspection tasks and solve known technical shortcomings.

[0083] The above embodiments are only used to illustrate some embodiments of the present invention and describe the technical features of the present invention, and do not limit the protection scope and scope of the present invention. Any modifications or equivalent arrangements that can be easily completed by anyone skilled in the art are within the scope of the present invention, and the protection scope of the rights of the present invention should be based on the scope of the patent application. [Explanation of symbols]

[0084] 1: Unmanned aerial vehicle 3: Computer 5: User control device NW:Network 11: Imaging device 12: Distance measurement and obstacle avoidance device 13: Processor 14: Flight and control systems 15: Positioning device 16:Communication equipment 100: Image schematic TB: Target bridge CL: Calibration curve FC: Ukigumo 200: Circulation operation schematic diagram SP: Calibration start point EP: Calibration end point 201: Image schematic TA: Object area AG:Angle RL: Reference line FD: Flight direction CP: Center point VL: Perpendicular 300: Feature map 302: Coordinate sequence 304: Starting point 306:End point 400: World coordinate system 402: Virtual coordinate system 404: Waypoint 406: Virtual Route P: Waypoint 500, 502, 504, 506: Waypoints 508: Flight Path 510:Environmental Map 600: Bridge inspection methods S601, S603: Step CS: Calibration stage S701, S702, S703, S704, S705: Operation IS: Patrol Phase S706, S707, S708, S709: Operation

Claims

1. A system for planning a flight path for an unmanned aerial vehicle equipped with an imaging device capable of capturing images above, such as flying while capturing images from below a bridge, comprising: The unmanned aerial vehicle takes an image of a target bridge above, and when the unmanned aerial vehicle detects that the image includes an obscured portion and an unobscured portion corresponding to a global positioning system, the unmanned aerial vehicle recognizes a calibration line corresponding to the target bridge in the image, and flies along the calibration line to perform a bridge inspection task; The calibration start point and the calibration end point of the calibration curve each correspond to a latitude and longitude coordinate in the real world. A system for planning flight paths for unmanned aerial vehicles.

2. Controlling the unmanned aerial vehicle to fly along the calibration curve includes: Calculating an included angle between the flight direction of the unmanned aerial vehicle and the calibration curve and a distance between the unmanned aerial vehicle and the calibration curve, and determining that the included angle between the flight direction and the calibration curve is equal to or less than an angle preset value; adjusting the flight direction by controlling the included angle between the flight direction and the calibration curve to be equal to or less than an angle preset value and the distance between the unmanned aerial vehicle and the calibration curve to be equal to or less than a distance preset value; The system of claim 1 , comprising:

3. The unmanned aerial vehicle includes: The system of claim 1 , further comprising: recording an actual flight path to create an environment map; and calculating a coordinate transformation matrix corresponding to the environment map and the real world based on the actual flight path and the calibration curve.

4. 4. The system of claim 3, wherein the actual flight path corresponds to a virtual coordinate system and the calibration curve corresponds to a world coordinate system, and the coordinate transformation matrix is ​​used to transform the virtual coordinate system to the world coordinate system.

5. The system of claim 3 , wherein the coordinate transformation matrix is ​​used to transform and integrate coordinate system orientation and coordinate system scale.

6. The unmanned aerial vehicle includes: Flying based on a plurality of virtual inspection routes and the environmental map; recording the actual flight path flown by the unmanned aerial vehicle to subsequently generate the map of the environment; The system of claim 3 , further comprising: a coordinate transformation matrix for positioning the unmanned aerial vehicle to calculate actual coordinate values ​​corresponding to each of a plurality of inspection images corresponding to the target bridge.

7. The unmanned aerial vehicle includes: The system according to claim 6 , wherein a plurality of real-world inspection routes are transformed into the virtual inspection route by an inverse coordinate transformation matrix generated based on the coordinate transformation matrix.

8. The system of claim 3 , wherein the map of the environment is created using a plurality of point clouds corresponding to the images.

9. The system of claim 3 , wherein the environmental map records at least one or a combination of shooting parameters, feature maps, coordinates, and timestamps corresponding to each of a plurality of locations.

10. A system for planning a flight path for an unmanned aerial vehicle equipped with an imaging device capable of capturing images above, such as flying while capturing images from below a bridge, comprising: The unmanned aerial vehicle takes an image of a target bridge above, and when the unmanned aerial vehicle detects that the image includes a bridge bottom portion and a sky portion, the unmanned aerial vehicle recognizes a calibration line corresponding to the target bridge in the image, and flies along the calibration line to perform a bridge inspection task; The calibration start point and the calibration end point of the calibration curve each correspond to a latitude and longitude coordinate in the real world. A system for planning flight paths for unmanned aerial vehicles.

11. A bridge inspection method for use with an unmanned aerial vehicle, the unmanned aerial vehicle including an upward-facing camera and a processor, the upward-facing camera being used to capture an image corresponding to a target bridge based on a tilt angle, the bridge inspection method comprising: Recognizing a calibration curve corresponding to the target bridge in the image; Controlling the unmanned aerial vehicle to fly along the calibration curve to perform a bridge inspection task based on the image, wherein a calibration start point and a calibration end point of the calibration curve each correspond to a latitude and longitude coordinate in a real world; A bridge inspection method including:

Images