Waypoint correction device and waypoint correction method

Abstract

This disclosure provides a waypoint correction device and a waypoint correction method. [Solution] A waypoint correction device that acquires real-time video from an unmanned aerial vehicle (UAV) in response to the UAV's positioning signal being located at a planned waypoint. The waypoint correction device calculates the feature point distribution in the real-time video based on the real-time video. The waypoint correction device generates a viewpoint adjustment command based on the feature point distribution and controls the UAV to rotate in place based on the viewpoint adjustment command and acquire the adjusted real-time video. The waypoint correction device generates a corrected flight path based on a plurality of 3D feature points, the adjusted real-time video, and a sampling count threshold and controls the UAV to move from its actual position to the planned waypoint based on the corrected flight path.

Description

Technical Field

[0001] The present invention relates to an unmanned aerial vehicle applicable to facility patrol inspection and a waypoint correction method. Specifically, the present invention relates to an unmanned aerial vehicle and a waypoint correction method capable of performing waypoint correction of an unmanned aerial vehicle by a viewpoint positioning method.

Background Art

[0002] Recently, with the rapid development of unmanned aerial vehicle (Unmanned Aerial Vehicle; UAV, also called unmanned aircraft or drone) technology, applications related to unmanned aerial vehicles have been proposed in large numbers one after another.

[0003] Furthermore, since unmanned aerial vehicles have cost and speed advantages, performing facility patrol inspection tasks (for example, safety inspection of bridge structures) using unmanned aerial vehicles is one of their important applications. When an unmanned aerial vehicle performs a facility patrol inspection task, there may be obstacles at the inspection site, or there may be cases where the unmanned aerial vehicle needs to take pictures at close range (for example, bridge inspection, building inspection, radio tower inspection).

Summary of the Invention

Problems to be Solved by the Invention

[0004] Taking bridge inspection as an example, a user can use a bridge member model including GPS information to plan a flight path to avoid encountering obstacles such as bridge piers and bridge handrails during flight. However, in the prior art, not all unmanned aerial vehicles are equipped with expensive ranging and obstacle avoidance modules or Real Time Kinematic (RTK) modules. Unmanned aerial vehicles usually use GPS as a navigation system, and the potential error in the positioning of GPS itself can reach 10 meters. Therefore, the unmanned aerial vehicle may collide with obstacles such as bridge columns due to GPS errors.

[0005] In light of this, providing waypoint correction technology that enables waypoint correction of unmanned aerial vehicles using viewpoint positioning is an urgent issue that the industry must address. [Means for solving the problem]

[0006] One object of the present invention is to provide a waypoint correction device. The unmanned aerial vehicle (UAV) includes a memory device and a processor, the processor being electrically connected to the memory device. The memory device is used to store a 3D map, the 3D map including a plurality of 3D feature points. The processor acquires real-time video from the UAV in response to acquiring a positioning signal for the UAV suitable for facility inspection. The processor calculates the feature point distribution in the real-time video based on the real-time video. The processor generates a viewpoint adjustment command based on the feature point distribution and controls the UAV to rotate in the local area based on the viewpoint adjustment command and acquire adjusted real-time video. The processor generates a corrected flight path based on the plurality of 3D feature points, the adjusted real-time video and a sampling threshold and controls the UAV to move from the actual position to a planned waypoint based on the corrected flight path.

[0007] Another object of the present invention is to provide a waypoint correction method for use in an electronic device. The electronic device stores a 3D map, the 3D map includes a plurality of 3D feature points. The waypoint correction method includes the steps of: acquiring real-time video from an unmanned aerial vehicle (UAV) in response to the positioning signal of the UAV being located at a planned waypoint; calculating the feature point distribution in the real-time video based on the real-time video; generating a viewpoint adjustment command based on the feature point distribution and controlling the UAV to rotate in the local area based on the viewpoint adjustment command and acquire adjusted real-time video; and generating a corrected flight path based on the plurality of 3D feature points, the adjusted real-time video and a sampling threshold, and controlling the UAV to move from its actual position to the planned waypoint based on the corrected flight path.

[0008] One object of the present invention is to provide a waypoint correction device. The waypoint correction device includes a storage device, a user interface, and a processor, the processor being electrically connected to the storage device and the user interface. The storage device is used to store a 3D map, the 3D map including a plurality of 3D feature points. The user interface is used by a user to control an unmanned aerial vehicle and includes a waypoint correction function activation option. The processor acquires real-time video from the unmanned aerial vehicle in response to the activation status of the waypoint correction function activation option. Based on the real-time video, the processor generates a viewpoint adjustment command and controls the unmanned aerial vehicle to rotate in the local area based on the viewpoint adjustment command and acquire adjusted real-time video. Based on the adjusted real-time video, the processor generates a corrected flight path and controls the unmanned aerial vehicle to move from its actual position to a planned waypoint based on the corrected flight path.

[0009] The waypoint correction technology provided in this disclosure (including at least the apparatus and method) actively analyzes the feature point distribution of real-time video to control an unmanned aerial vehicle (UAV) to adjust its viewpoint to acquire more feature points and more accurate real-time video. Next, the waypoint correction technology provided in this disclosure can generate a corrected flight path based on pre-stored 3D feature points and the adjusted real-time video, and control the UAV to move to the correct planned waypoint based on the corrected flight path. Furthermore, the waypoint correction technology provided in this disclosure can further consider the number of samplings and, if the projection error value result is unsatisfactory, adjust the viewpoint again to acquire new real-time video and make a decision. The waypoint correction technology provided in this disclosure can perform waypoint correction of an UAV using viewpoint positioning and does not require the installation of expensive modules, thus overcoming the shortcomings of the prior art.

[0010] The detailed technology and embodiments of the present invention will be described below with reference to the drawings, so that those skilled in the art can understand the technical features of the invention protected by the claims. [Brief explanation of the drawing]

[0011] [Figure 1] This is a schematic diagram showing the architecture and environment in several embodiments. [Figure 2A] This is a schematic diagram illustrating the inspection operation in several embodiments. [Figure 2B] This is a schematic diagram illustrating the inspection operation in several embodiments. [Figure 3A] This is a schematic diagram showing 3D maps in several embodiments. [Figure 3B] This is a schematic diagram illustrating the characteristic features of several embodiments. [Figure 4] This is a schematic diagram illustrating feature point matching in several embodiments. [Figure 5] This is a schematic diagram illustrating matching in several embodiments. [Figure 6]This is a schematic diagram illustrating the operation in several embodiments. [Figure 7] This is a schematic diagram illustrating the calibration operation in several embodiments. [Figure 8] This is a flowchart showing the waypoint modification method of the second embodiment. [Figure 9] This is a schematic diagram showing the user interface of several embodiments. [Modes for carrying out the invention]

[0012] The waypoint correction apparatus and method provided by the present invention will be described below with reference to embodiments. However, these embodiments do not limit the scope of the present invention to any environment, application, or method described in these embodiments. Therefore, the description of embodiments is for the purpose of illustrating the object of the present invention and does not limit the scope of the present invention. In the following embodiments and drawings, elements not directly related to the present invention have been omitted, and the sizes of each element and the size ratios between elements are illustrative and do not limit the scope of the present invention.

[0013] First, as shown in Figure 1, in the application environment of this disclosure, an unmanned aerial vehicle (UAV) equipped with one or more cameras captures real-time video 100 corresponding to a target object TO (e.g., a bridge) based on multiple waypoints and performs a patrol operation of the target object TO. This disclosure enables the unmanned aerial vehicle (UAV) to perform waypoint correction operations using a waypoint correction device (WCD).

[0014] For easier understanding, please refer to Figures 2A and 2B. As shown in Figure 2A, the unmanned aerial vehicle (UAV) receives a command for the inspection target object TO from the waypoint correction device (WCD), flies along a pre-planned route, and its inspection mission includes capturing real-time video at waypoints A1, A2, and A3, each waypoint accompanied by GPS coordinates and direction information.

[0015] In this embodiment, the flight path FR1 from waypoint A1 to waypoint A2 is a low-risk path because it is an open area, and the flight path FR2 from waypoint A2 to waypoint A3 is a high-risk path because it may pass through the target object TO.

[0016] FIG. 2B is a schematic diagram showing that the unmanned aerial vehicle UAV flew to an incorrect waypoint due to GPS error before flying on the high-risk path. In this embodiment, waypoint A2 is the correct waypoint for the unmanned aerial vehicle UAV to pass through the target object TO, and video A2P is the video acquired by the unmanned aerial vehicle UAV at waypoint A2. However, as shown in FIG. 2B, due to the positioning error of GPS, the actual position of the unmanned aerial vehicle UAV may be located at an incorrect waypoint A2', and furthermore, the unmanned aerial vehicle UAV may acquire an incorrect video A2P' at the incorrect waypoint A2'. If the flight mission is continued and executed at the incorrect waypoint, the unmanned aerial vehicle UAV may collide with the target object TO. The present disclosure can solve the above problems by executing a waypoint correction operation corresponding to the unmanned aerial vehicle UAV through the waypoint correction device WCD.

[0017] The first embodiment of the present invention is a waypoint correction device WCD, and its configuration schematic diagram is depicted in FIG. 1. The waypoint correction device WCD includes a storage device 11 and a processor 13, and the processor 13 is electrically connected to the storage device 11.

[0018] In some embodiments, as shown in FIG. 1, the unmanned aerial vehicle UAV includes an imaging device 21, a processor 23, a flight and control device 24, a positioning device 25, and a communication device 26.

[0019] In some embodiments, the imaging device 21 includes an inspection imaging device and a rotation mechanism (e.g., a gimbal).

[0020] In some embodiments, the flight and control device 24 includes a power unit (e.g., rotor or fixed wing), inertial measuring devices (e.g., accelerometer and gyroscope), and environmental sensing devices (e.g., barometer, thermometer, magnetometer, etc.). The positioning device 25 may be any device having a positioning function, such as a Global Positioning System (GPS) or a WiFi positioning device. The communication device 26 includes an antenna device.

[0021] Processor 13 and processor 23 may be various processing units, central processing units (CPUs), microprocessors, or other computing devices known to those skilled in the art. The storage device 11 may be memory, a Universal Serial Bus (USB) drive, a hard disk, an optical disc, a USB memory, or any other storage medium or circuit having equivalent functionality known to those skilled in the art.

[0022] Note that Figure 1 is for illustrative purposes only, and this disclosure does not limit the number of unmanned aerial vehicles (UAVs) that can be connected to a waypoint correction device (WCD).

[0023] In some embodiments, the unmanned aerial vehicle (UAV) can establish a communication connection (e.g., communication link CN) with a waypoint correction device (WCD) via an antenna in the communication device 26 and transmit relevant data (e.g., real-time video) and control signals.

[0024] In some embodiments, the waypoint correction device (WCD) may be installed on a ground-side computer used by user C, and the unmanned aerial vehicle (UAV) can transmit and process relevant video content generated by the UAV via a network (NW) to the waypoint correction device (WCD).

[0025] Furthermore, in some embodiments, the waypoint correction device (WCD) may be directly installed in the unmanned aerial vehicle (UAV) and directly electrically connected to elements within the UAV. Specifically, the memory and processor within the waypoint correction device (WCD) may be directly installed in the UAV (for example, sharing the UAV's processor 23), and the UAV may directly perform all operations related to the waypoint correction device (WCD) in this disclosure.

[0026] In this embodiment, the memory device 11 of the waypoint correction device WCD is used to store a 3D map 3DM, which includes a plurality of 3D feature points (e.g., a point cloud).

[0027] In this disclosure, prior to performing a scheduled patrol mission, the Waypoint Correction Device (WCD) first uses an unmanned aerial vehicle to photograph target objects (i.e., patrol targets) and construct 3D maps (e.g., 3D models) to collect images and related information of the target objects and contribute to the patrol plan.

[0028] For example, the Waypoint Correction Device (WCD) can perform a 3D reconstruction operation on multiple inspection target images to generate 3D point cloud information. The 3D reconstruction method is carried out using Structure from Motion (SFM) technology. Next, the Waypoint Correction Device (WCD) generates a 3D map (3DM) based on the 3D point cloud information.

[0029] For easier understanding, please refer to the schematic 3D map diagram 200 shown in Figure 3A. In this example, the target object is a bridge, and the 3D map corresponding to the target object is composed of multiple point cloud PCs, each point cloud PC containing information such as 3D coordinate information, color (RGB), and feature values.

[0030] In some embodiments, the 3D map 3DM is generated by the following operation. First, the processor 13 acquires multiple depth images in 3D space. Next, the processor 13 combines the multiple depth images to generate multiple point clouds, which are displayed based on the local coordinate system. Finally, the processor 13 transforms the multiple point clouds into the world coordinate system based on a transition matrix to generate the 3D map 3DM.

[0031] In this embodiment, when the processor 13 determines that the unmanned aerial vehicle (UAV) has flown to a waypoint, it starts the waypoint correction operation. Specifically, in response to the positioning signal of the UAV being located at the planned waypoint, the processor 13 acquires real-time video from the UAV.

[0032] Furthermore, due to potential issues with obstruction and light, the real-time video footage captured by the unmanned aerial vehicle (UAV) may lack sufficient feature points, leading to a decrease in the accuracy of subsequent viewpoint positioning based on these feature points.

[0033] To solve this problem, in this embodiment, the processor 13 determines whether to adjust the shooting viewpoint of the unmanned aerial vehicle (UAV) by determining the feature points of the real-time video. Specifically, the processor 13 calculates the feature point distribution in the real-time video based on the real-time video. Next, the processor 13 generates a viewpoint adjustment command based on the feature point distribution and controls the UAV to rotate in place (on the spot) based on the viewpoint adjustment command and acquire the adjusted real-time video.

[0034] In some embodiments, the operation for generating the viewpoint adjustment command includes the following: First, the processor 13 selects a first block having the maximum number of feature points from among a plurality of blocks in the real-time video based on the feature point distribution. Next, the processor 13 generates the viewpoint adjustment command based on the first block. The viewpoint adjustment command is used to control the unmanned aerial vehicle (UAV) to adjust the shooting viewpoint in the direction corresponding to the first block and acquire the adjusted real-time video.

[0035] For easier understanding, please refer to Figure 3B. As shown in Figure 3B, the processor 13 divides the real-time video 300 into four regions (up, down, left, and right) according to the diagonal DA, counts the number of matching feature points FP in each region, and then rotates the unmanned aerial vehicle (UAV) by θ degrees (e.g., 45 degrees) toward the region with the most feature points to acquire real-time video from a new viewpoint. In this embodiment, since most of the matching feature points FP in the real-time video 300 are in the left region, the UAV is rotated to the left by θ degrees to acquire real-time video 301 from a new viewpoint.

[0036] Finally, the processor 13 performs waypoint correction operations based on the adjusted real-time video and the feature point matching results of the plurality of 3D feature points. Specifically, the processor 13 generates a corrected flight path based on the plurality of 3D feature points, the adjusted real-time video, and a sampling threshold, and controls the unmanned aerial vehicle (UAV) to move from its actual position to a planned waypoint (i.e., the corrected waypoint) based on the corrected flight path.

[0037] In this disclosure, the processor 13 can set a sampling threshold (for example, 3 times) in advance. If the projection error value calculated by the processor 13 is outside the projection threshold, the processor 13 controls the unmanned aerial vehicle (UAV) again to capture real-time video from a new viewpoint (i.e., samples again), and continues until the sampling threshold is reached.

[0038] In some embodiments, generating the corrected flight path includes the following operations: First, in response to acquiring the adjusted real-time image from the unmanned aerial vehicle (UAV), the processor 13 performs a feature point matching operation based on the adjusted real-time image and the plurality of 3D feature points of the 3D map to generate a plurality of first feature point pairs and a first projection error value corresponding to the adjusted real-time image. Next, in response to the first projection error value being lower than the projection threshold or the number of resamplings being greater than the sampling threshold, the processor 13 calculates the actual position of the UAV based on the plurality of first feature point pairs. Finally, the processor 13 generates the corrected flight path based on the actual position and controls the UAV to move from the actual position to the planned waypoint based on the corrected flight path.

[0039] In some embodiments, the feature point matching operation includes the following: First, the processor 13 calculates a plurality of planar feature points in the adjusted real-time image. Next, the processor 13 compares the plurality of planar feature points and the plurality of 3D feature points to generate a plurality of first feature point pairs, each of which includes one of the plurality of planar feature points and one of the plurality of 3D feature points.

[0040] The goal of feature point matching is to pair two feature point sets from different sources and find feature point pairs that form a set. For example, the Brute-Force Matcher method can be used to compare whether feature points in two different feature point sets correspond to the same location in space. For easier understanding, please refer to Figure 4. Figure 4 is a schematic diagram 400 of feature point matching performed on multiple 3D feature points (e.g., point clouds) in the streaming video SI and 3D map of an unmanned aerial vehicle (UAV). In this embodiment, the processor 13 compares all feature points in the streaming video SI and 3D map of the UAV, and if two feature descriptors satisfy the matching condition, a feature point pair FPP is formed.

[0041] In some embodiments, generating the first projection error value includes the following operations: First, the processor 13 calculates a transition matrix based on the plurality of 3D feature points and corresponding plurality of planar feature points in the plurality of first feature point pairs, the transition matrix being used to convert the 3D coordinates in the 3D map to one of a plurality of planar coordinates in the adjusted real-time image. Next, the processor 13 converts the plurality of 3D feature points to the plurality of planar coordinates in the adjusted real-time image based on the transition matrix to generate a plurality of reprojection coordinates. Finally, the processor 13 compares the plurality of reprojection coordinates and the plurality of planar feature points to generate the first projection error value.

[0042] For easier understanding, please refer to Figure 5. This disclosure shows that the camera's orientation can be calculated using the PnP (Perspective-n-Point; PnP) algorithm. As shown in Figure 5, given n matching pairs of 3D feature points (3FP) and 2D feature points (2FP) (where n is 3 or greater) and a predefined camera internal orientation parameter, the orientation of the current streaming video can be calculated.

[0043] For example, camera orientation can be divided into two parts, position and orientation, and can be represented by the camera's external orientation parameters, the rotation matrix R and the translation vector T. R is a 3x3 matrix (r 11 ~r 33 ), where T is a 3x1 matrix (t1~t3). The arrays of 2D feature points are u, v, and the arrays of 3D points are x, y, z. The camera's internal orientation parameter is the focal length f in the two axes. x ,f y , and the principal image points c in the two axes x , c y It is composed of the following, and constitutes the following projection equations.

number

[0044] Furthermore, in the projection equation, the translation vector T can also be expressed using the rotation matrix R and the camera position C by the following equation.

number

[0045] Therefore, the camera position (X, Y, Z) can also be obtained by calculating it from the following rotation matrix and translation vector.

number

[0046] Furthermore, the reprojection error is given by the following formula: Error reproj The reprojection error is defined as the average difference between the 3D matching points projected onto the pixel plane via the projection matrix and the original 2D feature points. Based on the projection equation, an array of k sets of 3D matching points x, y, z can be transformed into an array of 2D points u', v', and the average error between these and the original array of 2D feature points u, v is the reprojection error. For example, a recommended threshold for the reprojection error is 2% of the longest side of the image, i.e., the threshold for a 1280x720 image is 25px.

number

[0047] In some embodiments, the processor 13 can control the unmanned aerial vehicle (UAV) to acquire new adjusted real-time images again based on the new viewpoint adjustment command. Specifically, in response to the first projection error value being higher than the projection threshold and the number of resamplings being lower than the sampling threshold, the processor 13 generates a new viewpoint adjustment command based on the feature point distribution of the adjusted real-time images and controls the UAV to acquire new adjusted real-time images again based on the new viewpoint adjustment command.

[0048] In some embodiments, to avoid excessive viewpoint adjustment, if the number of resamplings exceeds a sampling threshold, the processor 13 does not continue viewpoint adjustment and calculates the actual position based on the current information.

[0049] Specifically, the processor 13 generates a plurality of second feature point pairs and a second projection error value corresponding to the newly adjusted real-time image. Next, in response to the second projection error value being lower than the projection threshold, or the number of resamplings being greater than the sampling threshold, the processor 13 calculates the actual position of the unmanned aerial vehicle (UAV) based on the plurality of second feature point pairs and does not generate the viewpoint adjustment command.

[0050] In some embodiments, when the signal strength of the unmanned aerial vehicle (UAV) is low, the processor 13 spontaneously increases the frequency of acquiring real-time video from the UAV to improve the frequency of corrective flight path operations.

[0051] Specifically, in response to the signal intensity of the positioning signal being less than an intensity threshold, the processor 13 generates a control signal, which is used to control the frequency of acquiring the real-time video from the unmanned aerial vehicle (UAV), and the processor 13 also increases the frequency of receiving the real-time video from the UAV.

[0052] In some embodiments, when the unmanned aerial vehicle (UAV) is approaching a target object, the processor 13 spontaneously increases the frequency of acquiring real-time video from the UAV to improve the frequency of corrective flight path operations.

[0053] Specifically, in response to the distance of the planned waypoint from the target object being lower than a distance threshold, the processor 13 generates a control signal, which is used to control the frequency of acquiring the real-time video from the unmanned aerial vehicle (UAV) and to improve the frequency of receiving the real-time video from the UAV.

[0054] In some embodiments, the waypoint correction device (WCD) may further include a user interface operated by the user, which may be any device that can interact with the user. Specifically, the user interface is used to receive a flag value corresponding to the planned waypoint. In response to the positioning signal of the unmanned aerial vehicle (UAV) being located at the planned waypoint, the processor 13 determines, based on the flag value, whether or not to generate a control signal to control the acquisition of the real-time video by the UAV.

[0055] Using Figure 2A as an example, the user can configure the processor 13 to perform waypoint modification operations only at waypoints A2 and A3 by setting the flag values ​​for waypoints A2 and A3.

[0056] For easier understanding, please refer to the schematic diagram of operation in Figure 6. The operation of this disclosure is divided into a calibration phase and an inspection phase (for example, calibration phase CS and inspection phase IS in Figure 6). First, in calibration phase CS, the waypoint correction device WCD executes operation S601 to load the 3D map. Next, the waypoint correction device WCD executes operations S602 and S603 to perform a conversion between a virtual coordinate system (i.e., mapping coordinate system) and a real-world coordinate system (i.e., world coordinate system), and outputs 3D map information to be used as the base map for operation in inspection phase IS.

[0057] After the calibration phase CS is completed, the waypoint correction device (WCD) executes operation S604 to control the unmanned aerial vehicle (UAV) to fly to the waypoint. Next, the waypoint correction device (WCD) executes operation S605 to control the UAV to perform viewpoint sampling positioning. Then, the waypoint correction device (WCD) executes operation S606 to calculate the waypoint error.

[0058] Next, the Waypoint Corrector (WCD) performs operation S607 to determine whether the waypoint error is greater than a threshold, and if so, performs operation 608 to calibrate the Unmanned Aerial Vehicle (UAV). Otherwise, it terminates the operation.

[0059] For easier understanding, please refer to the schematic diagram of the calibration operation in Figure 7. First, the waypoint correction device WCD performs operation S701 to set the sampling counter to 1. Next, the waypoint correction device WCD performs operations S702, S703, and S704 to obtain feature point matching of the real-time video. Next, the waypoint correction device WCD performs operation S705 to calculate the orientation and reprojection error of the video.

[0060] Next, the Waypoint Corrector WCD performs operation S706 to determine whether the reprojection error is less than the projection threshold. If so, the Waypoint Corrector WCD performs operation S710 to calculate the final positioning coordinates. Otherwise, the Waypoint Corrector WCD performs operation S707 to determine whether the counter value is greater than the sampling threshold. If so, the Waypoint Corrector WCD performs operation S710 to calculate the final positioning coordinates.

[0061] Otherwise, the waypoint correction device (WCD) executes operations S708 and S709 to adjust the sampling viewpoint, increment the sampling counter value by 1, and then executes operation S702 again to re-perform the waypoint correction.

[0062] As can be seen from the above description, the Waypoint Correction Device (WCD) provided in this disclosure adjusts the viewpoint of the unmanned aerial vehicle (UAV) by actively analyzing the feature point distribution of real-time video, thereby acquiring more feature points and more accurate real-time video. Next, the Waypoint Correction Device (WCD) provided in this disclosure can generate a corrected flight path based on pre-stored 3D feature points and the adjusted real-time video, and control the UAV to move to the correct planned waypoint based on the corrected flight path. Furthermore, the Waypoint Correction Device (WCD) provided in this disclosure can further consider the number of samplings, and if the result of the projection error value is outside the projection threshold, it can adjust the viewpoint again to acquire new real-time video and make a decision. The Waypoint Correction Device (WCD) provided in this disclosure can perform waypoint correction of the UAV using viewpoint positioning and does not require the installation of expensive modules, thus solving the shortcomings of the prior art.

[0063] A second embodiment of the present invention is a waypoint correction method, the flowchart of which is shown in Figure 8. The waypoint correction method 800 is applied to an electronic device, for example, the waypoint correction device WCD described in the first embodiment. The electronic device stores a 3D map, the 3D map includes a plurality of three-dimensional feature points. The waypoint correction method 800 generates a corrected route through steps S801 to S807.

[0064] In step S801, the electronic device acquires real-time video from the unmanned aerial vehicle in response to the positioning signal of the unmanned aerial vehicle being located at a planned waypoint. Next, in step S803, the electronic device calculates the feature point distribution in the real-time video based on the real-time video.

[0065] Subsequently, in step S805, the electronic device generates a viewpoint adjustment command based on the feature point distribution and controls the unmanned aerial vehicle to rotate in place (on the ground) based on the viewpoint adjustment command and acquire real-time video after the adjustment.

[0066] Finally, in step S807, the electronic device generates a corrected flight path based on the plurality of 3D feature points, the adjusted real-time image, and the sampling count threshold, and controls the unmanned aerial vehicle to move from its actual position to a planned waypoint based on the corrected flight path.

[0067] In some embodiments, the step of generating the viewpoint adjustment command includes the step of selecting a first block having the maximum number of feature points from a plurality of blocks in the real-time video based on the feature point distribution, and the step of generating the viewpoint adjustment command based on the first block.

[0068] In some embodiments, the step of generating the corrected flight path includes, in response to acquiring the adjusted real-time image from the unmanned aerial vehicle, performing a feature point matching operation based on the adjusted real-time image and the plurality of 3D feature points of the 3D map to generate a plurality of first feature point pairs and a first projection error value corresponding to the adjusted real-time image; in response to the first projection error value being lower than a projection threshold or the number of resamplings being greater than the sampling threshold, calculating the actual position of the unmanned aerial vehicle based on the plurality of first feature point pairs; and generating the corrected flight path based on the actual position, wherein the electronic device controls the unmanned aerial vehicle to move from the actual position to the planned waypoint based on the corrected flight path.

[0069] In some embodiments, the feature point matching operation includes the steps of: calculating a plurality of planar feature points in the adjusted real-time image; and comparing the plurality of planar feature points with the plurality of 3D feature points to generate a plurality of first feature point pairs, each of which includes one of the plurality of planar feature points and one of the plurality of 3D feature points.

[0070] In some embodiments, the step of generating the first projection error value includes: calculating a transition matrix based on the plurality of 3D feature points and corresponding plurality of planar feature points in the plurality of first feature point pairs, the transition matrix being used to convert the 3D coordinates in the 3D map to one of a plurality of planar coordinates in the adjusted real-time image; converting the plurality of 3D feature points to the plurality of planar coordinates in the adjusted real-time image based on the transition matrix to generate a plurality of reprojection coordinates; and comparing the plurality of reprojection coordinates with the plurality of planar feature points to generate the first projection error value.

[0071] In some embodiments, the waypoint correction method 800 further includes the step of generating a new viewpoint adjustment command based on the feature point distribution of the adjusted real-time image in response to the first projection error value being higher than the projection threshold and the number of resamplings being smaller than the sampling threshold, and controlling the unmanned aerial vehicle to rotate in place (in place) based on the new viewpoint adjustment command and acquire a new adjusted real-time image again.

[0072] In some embodiments, the waypoint correction method 800 further includes the steps of generating a plurality of second feature point pairs and a second projection error value corresponding to the new adjusted real-time image, and in response to the second projection error value being lower than the projection threshold or the number of resamplings being greater than the sampling threshold, calculating the actual position of the unmanned aerial vehicle based on the plurality of second feature point pairs and not generating the viewpoint adjustment command.

[0073] In some embodiments, the waypoint correction method 800 generates a control signal in response to the signal intensity of the positioning signal being lower than an intensity threshold, wherein the control signal is used to control the frequency of acquisition of the real-time video in the unmanned aerial vehicle, and the electronic device further includes the step of increasing the frequency of reception of the real-time video from the unmanned aerial vehicle.

[0074] In some embodiments, the waypoint correction method 800 generates a control signal in response to the distance of the planned waypoint from a target object being lower than a distance threshold, wherein the control signal is used to control the frequency of acquisition of the real-time video in the unmanned aerial vehicle, and the electronic device further includes the step of increasing the frequency of reception of the real-time video acquired from the unmanned aerial vehicle.

[0075] In some embodiments, the electronic device further includes a user interface for receiving a flag value corresponding to the planned waypoint, and the waypoint correction method 800 further includes the step of determining whether to generate a control signal for controlling the acquisition of the real-time video by the unmanned aerial vehicle based on the flag value in response to the positioning signal of the unmanned aerial vehicle being located at the planned waypoint.

[0076] In addition to the above steps, the second embodiment can also perform all the operations and steps of the waypoint correction device WCD described in the first and second embodiments, having the same function and achieving the same technical effect. Those skilled in the art will be able to directly understand how the second embodiment performs these operations and steps based on the first embodiment, having the same function and achieving the same technical effect, and therefore will not be explained further.

[0077] Next, the following describes in detail a third embodiment of this disclosure. In short, in addition to the operations performed in the first and second embodiments, the third embodiment may further include a user interface for the waypoint correction device (WCD) (or a waypoint correction device installed on an unmanned aerial vehicle (UAV).

[0078] Specifically, the user interface is used by the user to control an unmanned aerial vehicle (UAV) and includes an option to activate a waypoint correction function. For example, the waypoint correction function activation option of the user interface may further include customizable settings for various thresholds, such as sampling count thresholds, distance thresholds, projection thresholds, intensity thresholds, etc.

[0079] Furthermore, in this embodiment, the waypoint correction device (WCD) acquires real-time video from the unmanned aerial vehicle (UAV) in response to the activation status of the waypoint correction function activation option. In this embodiment, the waypoint correction device (WCD) can perform the waypoint correction operations that are performed in the first and second embodiments.

[0080] For easier understanding, please refer to Figure 9. As shown in Figure 9, the user interface UI may include various setting interfaces and information display pages in the equipment inspection plan, such as the Waypoint Correction Enablement (WCE) option, route setting options, and multiple planned waypoints included in the route.

[0081] In this example, route 02 includes planned waypoints pt_001, pt_002, pt_003, pt_004, pt_005, pt_006, pt_007, pt_008, and pt_009, with each planned waypoint including related waypoint information, such as latitude and longitude data.

[0082] The user interface (UI) allows users to enable or disable the waypoint correction function for different routes. For example, in this example, routes 01 and 02 each have their own waypoint correction function activation options (WCEs), and the user can configure the waypoint correction function for routes 01 and 02 respectively using these WCEs.

[0083] The Waypoint Correction Function Activation Option (WCE) may further include enable / disable status setting options. For example, in this example, the Waypoint Correction Function Activation Option (WCE) further includes three options: "Waypoint Correction Function Enabled (System Default)", "Waypoint Correction Function Enabled (User Configurable)", and "Waypoint Correction Function Disabled". In this example, the "Waypoint Correction Function Enabled (System Default)" option activates the waypoint correction function, and the sampling count threshold and distance threshold are automatically set by the Waypoint Correction Device (WCD). The "Waypoint Correction Function Enabled (User Configurable)" option similarly activates the waypoint correction function, but allows the user to independently set the sampling count threshold and distance threshold. The "Waypoint Correction Function Disabled" option stops the waypoint correction function. This design allows users to set different waypoint correction function plans for different route configurations, further improving the safety of unmanned aerial vehicles during patrols.

[0084] As described above, the waypoint correction technology (including at least the apparatus and method) provided in this disclosure actively analyzes the feature point distribution of real-time video to control the unmanned aerial vehicle (UAV) to adjust its viewpoint to acquire more feature points and more accurate real-time video. Next, the waypoint correction technology provided in this disclosure can generate a corrected flight path based on pre-stored 3D feature points and the adjusted real-time video, and control the UAV to move to the correct planned waypoint based on the corrected flight path. Furthermore, the waypoint correction technology provided in this disclosure can further consider the number of samplings and, if the result of the projection error value is outside the projection threshold, adjust the viewpoint again to acquire new real-time video and make a decision. The waypoint correction technology provided in this disclosure can perform waypoint correction of an UAV using a viewpoint positioning method and does not require the installation of expensive modules, thus solving the shortcomings of the prior art.

[0085] The embodiments described above illustrate some embodiments of the present invention and explain its technical features, but do not limit the scope of protection or scope of the present invention. Any modifications or equivalent arrangements that are easily made by those skilled in the art fall within the scope claimed by the present invention, and the scope of protection of the present invention is in accordance with the claims. [Explanation of Symbols]

[0086] UAV: Unmanned aerial device WCD: Waypoint Correction Device 100: Real-time video TO: Target Object C:User NW: Network 11:Storage device 13: Processor 3DM: 3D Map CN: Communication Link 21: Imaging device 23: Processor 24: Flight and Control Systems 25: Positioning device 26: Communication equipment A1, A2, A3: Waypoints FR1, FR2: Flight paths A2': Incorrect waypoint A2P: Video A2P': Incorrect video 200: Schematic diagram of a 3D map PC: Point cloud 300: Real-time video 301: Real-time video DA: Diagonal 400: Schematic diagram of feature point matching FPP: Feature Point Pair SI: Streaming video 2FP: 2D feature points 3FP: 3D feature points CS: Calibration stage IS: Inspection phase S601~S608: Operation S701~S710: Operation 800: Waypoint Correction Method S801~S807: Process WCE: Waypoint Correction Function Activation Option UI: User Interface pt_001~pt_009: Planned Waypoints

Claims

1. Waypoint correction device, A storage device for storing a 3D map containing multiple 3D feature points, A processor electrically connected to the aforementioned storage device, Equipped with, Furthermore, the processor is The operation involves acquiring real-time video from the unmanned aerial vehicle in response to the positioning signal of the unmanned aerial vehicle being located at a planned waypoint, and Based on the real-time video, the operation of calculating the feature point distribution in the real-time video, The operation involves generating a viewpoint adjustment command based on the feature point distribution, controlling the unmanned aerial vehicle to rotate in the ground based on the viewpoint adjustment command, and acquiring real-time video after the adjustment; Based on the plurality of 3D feature points, the adjusted real-time video, and the sampling threshold, the operation of generating a corrected flight path and controlling the unmanned aerial vehicle to move from its actual position to the planned waypoint based on the corrected flight path, A waypoint correction device used to perform this task.

2. The operation that generates the aforementioned viewpoint adjustment command is: Based on the feature point distribution, the operation of selecting a first block having the maximum number of feature points from among multiple blocks in the real-time video, Based on the first block, an operation is performed to generate the viewpoint adjustment command, A waypoint correction device according to claim 1, including the following:

3. The operation to generate the aforementioned corrected route is: In response to acquiring the adjusted real-time video from the unmanned aerial vehicle, the operation involves performing a feature point matching operation based on the adjusted real-time video and the plurality of 3D feature points of the 3D map to generate a plurality of first feature point pairs and a first projection error value corresponding to the adjusted real-time video. An operation to calculate the actual position of the unmanned aerial vehicle based on the plurality of first feature point pairs in response to the first projection error value being lower than the projection threshold or the number of resamplings being greater than the sampling threshold, The processor generates the corrected flight path based on the actual position, and controls the unmanned aerial vehicle to move from the actual position to the planned waypoint based on the corrected flight path. A waypoint correction device according to claim 2, including the following:

4. The aforementioned feature point matching operation is performed as follows: The operation of calculating multiple planar feature points in the adjusted real-time video, The process involves comparing the plurality of planar feature points and the plurality of 3D feature points to generate the plurality of first feature point pairs, and each of the plurality of first feature point pairs includes one of the plurality of planar feature points and one of the plurality of 3D feature points. A waypoint correction device according to claim 3, including the following:

5. The operation for generating the first projection error value is: Based on the plurality of 3D feature points and corresponding plurality of planar feature points in the plurality of first feature point pairs, a transition matrix is ​​calculated, and the transition matrix is ​​used to convert the 3D coordinates in the 3D map to one of the plurality of planar coordinates in the adjusted real-time video. Based on the transition matrix, the operation involves converting the plurality of 3D feature points into the plurality of planar coordinates of the adjusted real-time video to generate a plurality of reprojection coordinates, The operation of generating a first projection error value by comparing the plurality of reprojection coordinates and the plurality of planar feature points, A waypoint correction device according to claim 3, including the following:

6. The aforementioned processor, The waypoint correction device according to claim 3, further performing an operation to generate a new viewpoint adjustment command based on the feature point distribution of the adjusted real-time image in response to the first projection error value being higher than the projection threshold and the number of resamplings being smaller than the sampling count threshold, and controlling the unmanned aerial vehicle to rotate in place based on the new viewpoint adjustment command and acquire a new adjusted real-time image again.

7. The aforementioned processor, The operation of generating a plurality of second feature point pairs and second projection error values ​​corresponding to the newly adjusted real-time video, In response to the second projection error value being lower than the projection threshold or the number of resamplings being greater than the sampling threshold, the actual position of the unmanned aerial vehicle is calculated based on the plurality of second feature point pairs, and no further viewpoint adjustment commands are generated. The waypoint correction device according to claim 6, further performing the following:

8. The aforementioned processor, The waypoint correction device according to claim 1, wherein, in response to the signal intensity of the positioning signal being lower than an intensity threshold, a control signal is generated, and the control signal is used to control the unmanned aerial vehicle to improve the acquisition frequency at which it acquires the real-time video, and the processor further performs an operation to improve the reception frequency at which it receives the real-time video from the unmanned aerial vehicle.

9. The aforementioned processor, The waypoint correction device according to claim 1, wherein, in response to the distance of the planned waypoint from the target object being lower than a distance threshold, a control signal is generated, and the control signal is used to control the unmanned aerial vehicle to acquire the real-time video at an increased acquisition frequency, and the processor further performs an operation to increase the reception frequency of receiving the real-time video from the unmanned aerial vehicle.

10. A waypoint correction method used in an electronic device that stores a 3D map containing multiple 3D feature points, The process of acquiring real-time video from the unmanned aerial vehicle in response to the positioning signal of the unmanned aerial vehicle being located at a planned waypoint, A step of calculating the feature point distribution in the real-time video based on the real-time video, The process involves generating a viewpoint adjustment command based on the feature point distribution, controlling the unmanned aerial vehicle to rotate in the ground based on the viewpoint adjustment command, and acquiring real-time video after the adjustment; The process involves generating a corrected flight path based on the plurality of 3D feature points, the adjusted real-time video, and the sampling threshold, and controlling the unmanned aerial vehicle to move from its actual position to the planned waypoint based on the corrected flight path. Waypoint modification methods including

11. The process of generating the aforementioned viewpoint adjustment command is as follows: A step of selecting a first block having the maximum number of feature points from a plurality of blocks in the real-time video based on the feature point distribution, A step of generating the viewpoint adjustment command based on the first block, A waypoint modification method according to claim 10, including the following:

12. The process of generating the corrected route is as follows: In response to acquiring the adjusted real-time video from the unmanned aerial vehicle, the process involves performing a feature point matching process based on the adjusted real-time video and the plurality of 3D feature points of the 3D map to generate a plurality of first feature point pairs and a first projection error value corresponding to the adjusted real-time video. A step of calculating the actual position of the unmanned aerial vehicle based on the plurality of first feature point pairs in response to the first projection error value being lower than the projection threshold or the number of resamplings being greater than the sampling threshold, The process includes generating the corrected flight path based on the actual position, and controlling the electronic device to move the unmanned aerial vehicle from the actual position to the planned waypoint based on the corrected flight path, A waypoint modification method according to claim 11, including the following:

13. The feature point matching process described above is: The process of calculating multiple planar feature points in the adjusted real-time video, The process involves comparing the plurality of planar feature points and the plurality of 3D feature points to generate the plurality of first feature point pairs, each of which includes one of the plurality of planar feature points and one of the plurality of 3D feature points, A waypoint modification method according to claim 12, including the following:

14. The step of generating the first projection error value is: A step in which a transition matrix is ​​calculated based on the plurality of 3D feature points and corresponding plurality of planar feature points in the plurality of first feature point pairs, and the transition matrix is ​​used to convert the 3D coordinates in the 3D map to one of the plurality of planar coordinates in the adjusted real-time video, Based on the transition matrix, the process involves converting the plurality of 3D feature points into the plurality of planar coordinates of the adjusted real-time video to generate a plurality of reprojection coordinates. A step of generating a first projection error value by comparing the plurality of reprojection coordinates and the plurality of planar feature points, A waypoint modification method according to claim 12, including the following:

15. The waypoint correction method according to claim 12, further comprising the step of generating a new adjusted viewpoint command based on the feature point distribution of the adjusted real-time image in response to the first projection error value being higher than the projection threshold and the number of resamplings being smaller than the sampling threshold, and controlling the unmanned aerial vehicle to acquire a new adjusted real-time image again based on the new adjusted viewpoint command.

16. A step of generating a plurality of second feature point pairs and a second projection error value corresponding to the newly adjusted real-time image, A step of calculating the actual position of the unmanned aerial vehicle based on the plurality of second feature point pairs in response to the second projection error value being lower than the projection threshold or the number of resamplings being greater than the sampling threshold, and not generating any further viewpoint adjustment commands, The waypoint modification method according to claim 15, further comprising:

17. The waypoint correction method according to claim 10, further comprising the steps of generating a control signal in response to the signal intensity of the positioning signal being lower than an intensity threshold, the control signal being used to control the unmanned aerial vehicle to improve the acquisition frequency at which it acquires the real-time video, and the electronic device improving the reception frequency at which it receives the real-time video from the unmanned aerial vehicle.

18. The waypoint modification method according to claim 10, further comprising the steps of generating a control signal in response to the distance of the planned waypoint from a target object being lower than a distance threshold, the control signal being used to control the unmanned aerial vehicle to acquire the real-time video at an increased rate, and the electronic device to receive the real-time video from the unmanned aerial vehicle at an increased rate.

19. The electronic device further includes a user interface for receiving a flag value corresponding to the planned waypoint, and the waypoint modification method is The waypoint correction method according to claim 10, further comprising the step of determining whether or not to generate a control signal for controlling the acquisition of real-time video by the unmanned aerial vehicle based on the flag value in response to the positioning signal of the unmanned aerial vehicle being located at the planned waypoint.

20. A storage device for storing a 3D map containing multiple 3D feature points, It is used by users to control unmanned aerial vehicles and includes a user interface with a waypoint correction function activation option, A processor electrically connected to the storage device and the user interface, Equipped with, Furthermore, the processor is In response to the activation status of the waypoint correction function activation option, the operation of acquiring real-time video from the unmanned aerial vehicle, Based on the aforementioned real-time video, a viewpoint adjustment command is generated, and the unmanned aerial vehicle is controlled to rotate in the ground based on the viewpoint adjustment command and acquire the adjusted real-time video; Based on the adjusted real-time video, the system generates a corrected flight path and controls the unmanned aerial vehicle to move from its actual position to a planned waypoint based on the corrected flight path. A waypoint correction device used to perform this task.

Images