Contour scanning using unmanned aerial vehicles
The UAV system autonomously navigates along contour paths to efficiently acquire images and generate 3D models of complex objects, addressing the inefficiencies of traditional human-controlled image acquisition methods.
Patent Information
- Application Number
- JP2024527391
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-11-10
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2041-11-10
AI Technical Summary
Acquiring images of a target object from a desired distance and with desired resolution and accuracy using traditional UAVs controlled by human pilots is tedious and inefficient.
A UAV system that autonomously determines multiple contour paths spaced apart from the scan target, acquires images at regular intervals along these paths, and generates a 3D model by associating a coordinate system with the target, adjusting speed based on illumination and blur thresholds, and using multiple cameras for navigation and image capture.
Enables efficient and accurate image acquisition of complex objects with varying geometries, allowing for the generation of high-resolution 3D models with improved coverage and reduced operational complexity.
Smart Images

Figure 2024544898000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to the field of unmanned aerial vehicles. [Background technology]
[0002] Unmanned aerial vehicles (UAVs), sometimes referred to as "drones," typically include one or more cameras for acquiring images of objects during flight. For example, such UAVs may be used to acquire images from vantage points that would otherwise be difficult to reach. Traditionally, UAVs have been controlled by ground pilots to acquire images of desired targets. However, acquiring images of the surface of a target object at a desired distance from the surface and with a desired resolution and accuracy of coverage can be a tedious task for a human pilot. Summary of the Invention
[0003] Some implementations include a UAV capable of determining multiple contour paths spaced apart from one another in at least one axis associated with the scanned object. For example, each contour path may be spaced apart from the scanned object's surface based on a selected distance. The UAV may determine multiple image acquisition locations for each contour path. The image acquisition locations may indicate locations where images of the scanned object's surface should be acquired. The UAV may navigate along the multiple contour paths based on the determined speed while acquiring images of the scanned object's surface based on the image acquisition locations. [Brief description of the drawings]
[0004] The detailed description is presented with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears, and the use of the same reference number in different figures indicates similar or identical items or features. [Figure 1] FIG. 1 illustrates an example system including an unmanned aerial vehicle (UAV) capable of acquiring images according to some implementations. [Diagram 2] 1A-1C show examples of determining contour paths and waypoints for a slice of a scanned object according to some implementations. [Figure 3A] 1A-1C illustrate examples of determining a contour path of a scanned object according to some implementations. [Figure 3B] 1A-1C illustrate examples of determining a contour path of a scanned object according to some implementations. [Figure 4A] 1A-1C illustrate examples of determining a contour path of a scanned object according to some implementations. [Figure 4B] 1A-1C illustrate examples of determining a contour path of a scanned object according to some implementations. [Diagram 5] FIG. 2 is a flow diagram illustrating an example process for scanning a scan target according to some implementations. [Figure 6] FIG. 1 is a flow diagram illustrating an example process for generating a 3D model of a scanned object according to some implementations. [Figure 7] A diagram showing an example configuration of the upper side of a UAV according to some implementations. [Figure 8] A diagram showing an example configuration of the bottom side of a UAV according to some implementations. [Figure 9] A diagram illustrating selected components of an example UAV according to some implementations. [Figure 10] FIG. 1 illustrates an example configuration of a controller including a computing device according to some implementations. [Figure 11] FIG. 2 illustrates select components of an example controller according to some implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0005] Some implementations herein are directed to techniques and arrangements for configuring a UAV to scan a three-dimensional (3D) scan target, such as a structure, building, bridge, pipeline, facility, site, scene, geographic feature, or any other object that may be designated as a scan target by an operator. For example, the UAV may be configurable to autonomously scan the scan target based on an initial indication of the scan target. The scan may include acquiring a series of images of the surface of the scan target indicated at a desired distance from the surface. The UAV may autonomously acquire images along a series of slices of the scan target in a complete and repeatable manner. The UAV may acquire scan targets having complex geometries, such as scan targets including recesses, uneven and sloping surfaces and features, openings, asymmetric geometries, and the like. In some cases, the acquired images may be used to generate a 3D model of the scan target or for any of a variety of other end uses.
[0006] As one example, the UAV may be configured to traverse a path corresponding to a plurality of virtual contours around the scan target at a selected distance (sometimes referred to herein as a baseline distance) from the surface of the scan target. The UAV may determine the virtual contours by dividing the scan target into a series of slices acquired across the axes of the scan target. As one example, the UAV may associate a coordinate system (such as an x, y, z coordinate system) with the scan target and divide the scan target into a plurality of slices across at least one axis of the coordinate system. The UAV may determine a contour for each slice based on a selected distance and a detected position of the surface of the scan target (also known as a zero contour). Additionally, the UAV may determine a waypoint and a point of interest for each contour. Each waypoint may be an image acquisition location where the UAV acquires an image of the surface of the scan target. The UAV may acquire images of the scanned object in an orderly manner by moving along multiple contours determined for the scanned object in an orderly, sequential manner, such as by acquiring an image along a first contour of a first slice, then moving to the next adjacent slice and acquiring an image along the contour determined for that slice, and continuing to move to the next adjacent slice until all desired portions of the scanned object have been acquired.
[0007] Each slice may be equivalent to a two-dimensional (2D) contour in the plane at the location in space of that particular slice. The UAV may aim a camera at the target and traverse the contour as a path of movement to acquire an image of the target's surface corresponding to the location of the slice. For example, the contour may essentially follow the shape of the target's surface at the location in space of that slice and may form a path for the UAV to traverse at a selected distance from the target's surface while acquiring an image of the target's surface. The UAV may select a pattern for traversing the contour that allows the UAV to optimally move at a constant speed while acquiring images at regular intervals.
[0008] The distance between slices may be determined based at least on the field of view of the camera mounted on the UAV used to acquire the images, as well as a selected distance of the path of travel from the surface of the scanned object, and a desired level of side overlap between adjacent slices of the acquired images. The distance of the camera from the surface of the scanned object may be selected based at least on the capabilities of the camera on the UAV and the desired level of detail in the acquired images. For example, the desired level of detail, such as resolution, sharpness, exposure time, etc., may depend at least in part on the intended subsequent use of the acquired images.
[0009] As will be further described below, depending in part on the configuration of the scan target, the UAV may be configured to traverse the determined virtual contour around the scan target using any of a variety of patterns. As one example, the UAV may move from the contour in one slice to the contour in the next slice using a lawnmower pattern, moving in the opposite direction in each alternating slice in a back and forth pattern. As another example, the UAV may perform a trajectory of the scan target, moving above / below or left / right for the next adjacent slice (e.g., depending on which axis the slice is acquired on) when it reaches the end of the first slice and moves to the location of the start of the next adjacent slice. For example, the UAV may traverse the slices as multiple trajectories (e.g., circles, ellipses, etc.) around the scan target by making a first trajectory at the location of the first slice, moving to the location of the next slice, making another trajectory of the scan target along the contour determined for that slice, etc. As yet another example, the UAV may traverse the slice as a series of adjacent vertical pillars, e.g., moving vertically along the length of the scanned object on a first vertical slice of the scanned object, then moving to an adjacent slice and moving along that slice as another vertical pillar.
[0010] Further, in some examples, depending on the configuration of the scanned object, additional slices may be determined along a second axis (e.g., perpendicular to the first axis) to obtain additional images of the scanned object. For example, assume that a first series of slices of the scanned object is obtained by slicing the scanned object across the z-axis to obtain images of the side of the scanned object. The scanned object may be further sliced across the x-axis or y-axis to obtain images of the top and / or bottom of the scanned object, etc. Furthermore, additional slices may be obtained across a third axis or some combination of axes, as needed. Furthermore, while the examples described herein describe multiple optimal slicing techniques and slicing patterns, numerous variations in slicing a particular scanned object and selecting possible patterns for traversing the determined contour for slicing will be apparent to one of ordinary skill in the art having the benefit of the disclosure herein.
[0011] In some implementations herein, the UAV may acquire images at regular intervals, determining a structured continuous path that may be traversed at a constant speed, etc., while acquiring the regularly spaced images. The speed may possibly be determined based on an acceptable threshold amount of motion blur in the captured images, which may also depend in part on the intended use of the captured images and the level of detail required. For example, the further away from the surface to be scanned that the images are acquired, the faster the speed of the UAV can be while acquiring successive images of the surface as it moves along the contoured moving path. On the other hand, if a detailed close-up image of the surface is desired, the UAV may slow down or stop to acquire images such that motion blur is eliminated.
[0012] For purposes of illustration, several example implementations are described for configuring a UAV to autonomously scan a target and acquire images of the target. However, implementations herein are not limited to the specific examples provided, and may be extended to other types of targets, other types of vehicles, other types of flight path planning techniques, other types of scans, other types of image acquisition, etc., as would be apparent to one of ordinary skill in the art in light of the disclosure herein.
[0013] 1 illustrates an example system 100 including an unmanned aerial vehicle (UAV) 102 capable of acquiring images according to some implementations. In this example, the UAV 102 can be in communication with a controller 104. The controller 104 may, in some cases, include a computing device. As one example, the controller 104 may include a mobile device, such as a mobile phone, a tablet computing device, a wearable device, a laptop computer, etc., that can wirelessly communicate with the UAV 102 directly or, alternatively, indirectly via another device, such as a network.
[0014] One or more processors (not shown in FIG. 1 ) on board the UAV 102 may be configured with executable instructions to receive images from at least one camera attached to the UAV 102. In the illustrated example, the UAV 102 includes multiple cameras, such as a first camera 106 and multiple second cameras 108. For example, the first camera 106 may include a lens with a longer focal length and may include a higher resolution image sensor than the second camera 108. The second camera 108 may have a shorter focal length, a wider field of view (FOV), and a lower resolution image sensor than the first camera 106. The first camera 106 may be mounted on a gimbal 110 to enable the first camera 106 to be directed toward a desired target object, such as the scan target 111 in this example, or other scan targets. In some cases, the second camera 108 may provide images that the UAV 102 can use for various purposes, such as navigation, position determination, distance determination, stereo imaging, obstacle avoidance, tracking, etc.
[0015] The UAV 102 can communicate with the controller 104 via wireless communication or the like. The controller 104 may be controlled by a user 112 and may be configured for two-way communication with the UAV 102 via any of a variety of types of wireless communication technologies. By way of example, the controller 104 may communicate via a variety of types of wireless protocols and frequencies, such as via a Wi-Fi network, a BLUETOOTH® wireless link, cellular radio, direct ISM band communication, or any other suitable wireless communication. For example, 900 MHz, 2.4 GHz, and 5.8 GHz are the most common wireless frequencies used for two-way communication with UAVs, although implementations herein are not limited to any particular communication type, frequency, or protocol.
[0016] Additionally or alternatively, the UAV 102 may communicate with one or more networks 113, such as via any of the aforementioned wireless communication types or any other type of wireless communication technology. As one example, the network 113 may include wireless access points, cellular radio towers or other cellular transceivers, short-range wireless transceivers, etc., to enable the UAV 102 to connect to and communicate via the one or more networks 113.
[0017] The UAV 102 includes a body 114 and one or more propulsion devices 116. In this example, a first set of second cameras 108 is mounted on the upper side of the body 114 and a second set is mounted on the lower side of the body 114. Additionally, the first camera 106 may include a fixed focus lens, or alternatively, may include a lens capable of optical zoom. The gimbal 110 allows the first camera 106 to be aimed and focused on an object without having to rotate the UAV 102 to point directly at the desired object. The UAV 102 may acquire images by the first camera 106 and may transmit at least a portion of the acquired images as images 118 to the controller 104. Additionally, in some implementations, at least a portion of the images acquired by the second camera 108 may be transmitted as part of the images 118. In some examples, the transmitted images 118 may be lower resolution images compared to the resolution of the images acquired by the UAV 102 to enable faster wireless transmission to the controller 104, etc.
[0018] In some cases, the UAV 102 may use images acquired by the UAV 102 to generate 3D model information that is onboard the UAV 102, and at least a portion of the 3D model information may be transmitted to the controller 104 as model information 120. The controller 104 may receive images 118 (which may include video or still images) from the UAV 102 via a wireless communication link. Based on the received images 118, the user 112 may use a display 124 associated with the controller 104 to view the field of view acquired by the UAV 102 and / or a 3D model of the scanned object 111, which may be generated at least in part by the UAV 102 and provided as 3D model information 120.
[0019] In some examples, the user 112 may use the controller 104 to select the scan target 111 in order to instruct the UAV 102 to perform a scan of the scan target 111. Additionally, the user 112 may use the controller 104 to issue other conventional commands, e.g., "take off," "land," "follow," etc., to the UAV 102 via one or more virtual controls presented in a graphical user interface (GUI) and / or one or more physical controls, such as joysticks, buttons, etc., included in the controller 104 (not shown in FIG. 1 ). The controller 104 may thus enable the user 112 to provide manual control inputs to manually control the UAV 102 and / or to instruct the UAV 102 to operate autonomously. An example controller 104 is further shown and described below with respect to FIGS. 10 and 11 .
[0020] In some examples, the controller 104 and / or the UAV 102 may be able to communicate with one or more service computing devices 126, or other suitable computing devices, such as other user computing devices, via one or more networks 113. The one or more networks 113 may include any suitable network or other communications technology, including a wide area network (WAN) such as the Internet, a local area network (LAN) such as an intranet, a wireless network such as a cellular network or other wireless communication, a local wireless network such as Wi-Fi, a short-range wireless communication such as BLUETOOTH, a wired network including optical fiber and Ethernet, any combination of these, or any other suitable communications network or other communications technology.
[0021] In some cases, the service computing device 126 may be located remotely from the controller 104 and / or the UAV 102, such as located at a cloud computing location, a data center, a server farm, or the like. The service computing device 126 may include a management program 128 that may be executed to communicate with the controller 104 and / or the UAV 102, such as to receive images 132 for storage in the database 130. The images 132 may include the images 118 described above and / or may include a complete set of images of the scan object 111 acquired by the UAV 102 during the scan of the scan object 111. For example, the complete set of images at a specified resolution may be uploaded to the service computing device 126 over time, such as after the UAV 102 completes the scan of the scan object 111. Additionally, the database 130 may include 3D model information 134. For example, the 3D model information may include the 3D model information 120 transmitted by the UAV 102. Additionally or alternatively, the management program 128 may execute one or more 3D modeling programs using the images 132 received from the UAV 102 and the associated location information received to generate a high-resolution 3D model of the scanned object 111 that can be textured using the acquired images.
[0022] Additionally, the management program 128 may perform other functions to manage the images 132 and other information received from the controller 104 and / or from the UAV 102. In some cases, the service computing device 126 may include a web application (not shown in FIG. 1 ) that allows a user 112 associated with the controller 104 to access the images 132 in the database 130, the 3D model information 134, and / or other information related to the UAV 102, etc.
[0023] In some examples, one or more processors on board the UAV 102 may be configured by program code or other executable instructions to perform the autonomous operations described herein. For example, the one or more processors may control the UAV 102 to navigate along an intended flight path while also performing other operations described herein, such as generating or accessing an initial lower resolution model of the scan object 111, determining multiple slices of the scan object 111 based on the initial model, determining the contours of each slice, and determining waypoints for acquiring images of the surface of the scan object 111. The UAV 102 may determine a scan plan for navigating the UAV 102 along a multiple contour path, such as at a constant speed, to acquire regularly spaced images at regular intervals while also avoiding obstacles.
[0024] The initial model of the scanned object 111 may be updated in real-time during the scan based on additional image information acquired by the UAV 102 during the scan. Additionally or alternatively, in some cases, a controller 104 or a service computing device 126, remote from and in communication with the UAV 102, may provide instructions to a processor on board the UAV 102, such as to assist or manage one or more of the aforementioned operations. For example, rather than having the UAV create an initial model of the scanned object using a distance determination, the initial model may be received by the UAV 102 from the controller 104 or the service computing device 126.
[0025] To initiate a scan, the user 112 may first indicate the scan target 111 or a portion of the scan target 111, such as by making one or more inputs to a user interface presented on the controller 104 or other computing device that can provide input information to the UAV 102. As one example, the user 112 may manually navigate the UAV 102 to obtain an image of the scan target, or the user 112 may create a polygon or other 2D shape on the image of the scan target presented in the user interface to indicate the scan target. Alternatively, the user may specify a bounding volume around the image of the scan target in the user interface, may specify a bounding area based on three or more reference points (e.g., "pillars"), or the like, or may specify the scan target to the UAV 102 using any of a number of other techniques. For example, the user may employ a technique for specifying the scan target that is determined to be most effective, such as depending on the shape of the scan target, the type of scan desired, the portion of the target desired to be scanned, or the like.
[0026] Based on the received indication of the scan target 111, the UAV 102 may correlate the indication of the scan target 111 with one or more positions in space, such as based on a global coordinate system, a correlation with a navigation coordinate system, or by various other techniques. For example, the UAV 102 may autonomously perform an initial coarse scan of the scan target 111 by performing an initial scan using, for example, a range sensor, or the like, to determine the location of the surface of the scan target in 3D space. As one example, the range sensor may be provided by one or more of the second cameras 108, each of which includes an image sensor. For example, two or more arrays of the second cameras 108 may be configured for stereoscopic imaging and may be used to determine distances from the UAV 102 to various points on the surface of the scan target 111. The second camera 108 may be used by the UAV 102 to enable autonomous navigation and may provide one or more lower resolution (e.g., compared to images received from the high resolution image sensor of the first camera 106) images that may be used to determine the distance from the UAV 102 to the surface of the scanned object 111. If the second camera 108 includes one or more stereo imaging pairs, the UAV 102 may determine the disparity between two related images of the same surface using input from the second camera 108 to determine the relative distance from the UAV 102 to one or more points on the surface of the scanned object 111. Alternatively, images from one or more monocular second cameras 108 taken from different angles may be used to determine the position of the scanned object's surface in 3D space, such as relative to the known position of the UAV 102, and / or various other distance sensing and 3D reconstruction techniques may be used.
[0027] As one example, the UAV 102 may fuse range images from the stereo pair of the second camera 108 and / or the wide-baseline multi-view stereo (MVS) pair of the second camera 108 into a volumetric Signed Distance Function (SDF) that can provide a surface position estimate containing a plurality of points. For example, an occupancy map may be determined first, and then a complete SDF model may be determined for the scanned object 111 that contains the positions of a plurality of points in 3D space. The plurality of points may indicate the positions of various points on the surface 136 of the scanned object 111.
[0028] As another alternative, the UAV 102 may sample the surface 136 of the object 111 for contours at the surface 136 and may extend these zero contours based on a function of the normal of each zero contour, which may include a smoothing step to determine a contour path at a selected distance from the zero contour (i.e., the surface of the object 111). Spline fitting may be one technique for this alternative technique. Furthermore, performing an initial sampling at a greater distance from the surface 136 of the object 111 may be more efficient for quickly generating an overall lower resolution 3D model of the object 111, but may limit the contour details of smaller surfaces of the object 111, resulting in poor image acquisition during detailed scanning of the object 111, such as when tilted surfaces are present.
[0029] As previously mentioned, the initial SDF model may be a low-resolution 3D model that typically includes a number of points in 3D space indicating the location of some points on the surface 136 of the scanned object 111, and may typically include the location of major edges 138 of the scanned object 111 in 3D space. The SDF model may be referred to as a lower-resolution 3D model with lower accuracy because the scanned object 111 has not yet been imaged or otherwise scanned from a sufficiently diverse set of viewpoints and / or a sufficiently close distance for greater accuracy. For example, the closer the UAV 102 flies to the surface as the UAV 102 continues to fuse the stereo pair range images into the model, the more accurate the estimate of the surface's location and shape of the surface in the SDF model will be.
[0030] An initial scan may be performed by the UAV 102 to generate a low-resolution 3D model of the scanned object. During the initial scan, the UAV 102 may autonomously image one or more surfaces of the scanned object 111. The UAV 102 may fly in close proximity to the scanned object 111 and acquire additional images during the initial scan or later when performing a complete scan of the scanned object 111, dynamically improving the completeness and / or resolution of the lower-resolution 3D model in real time. For example, the lower-resolution 3D model may include a set of multiple points in 3D space, the locations of the points corresponding to the surface of the scanned object 111 determined based on distance measurements.
[0031] The UAV 102 may use the lower resolution 3D model to generate a scan plan for performing a scan (i.e., acquiring high resolution images) of the scan object 111. For example, the scan may include acquiring a series of images of the surface 136 of the scan object 111 from a selected distance (e.g., a selected ground resolution) between the UAV 102 and the scan object 111 to acquire images with a desired resolution, detail, overlap, etc. For example, determining the scan plan may include slicing the scan object 111 into multiple slices 140 along a selected axis, a centerline of the scan object, a body of the scan object, etc., as described further below. In the illustrated example, it is assumed that the UAV 102 has divided the scan object 111 into multiple slices 140(1)-140(14) acquired along a Z-axis of a coordinate system 142 determined for the scan object 111, as described further below.
[0032] Additionally, the UAV 102 may select a distance D from the surface 136 of the scan target 111 to use as a baseline distance. In some cases, the distance D may be a predetermined default value designated for the particular type of scan to be performed based on the desired level of detail of the images acquired, the intended use of the scan results, etc. Additionally or alternatively, the distance D may be specified by the user 112.
[0033] Additionally, the UAV 102 may determine an overlap percentage O and a side lap percentage S for images acquired during a scan. For example, the overlap O may be a percentage of overlap of successive images acquired in a direction of movement of the UAV when acquiring images during a scan. Side lap (also known as lateral overlap or side overlap) may refer to an amount of overlap between images of adjacent contour slices 140, i.e., an amount of side overlap between an image acquired along a second slice 140(2) and an image acquired along a first slice 140(1) adjacent to the second slice 140(2). In some cases, the overlap and side lap may be default values predetermined for performing a particular type of scan and / or may be specified by the user 112, etc.
[0034] Based on the camera FOV of the camera used to acquire images during the scan (e.g., the FOV of the first camera 106 in the example of FIG. 1), the UAV 102 may determine a waypoint baseline Bw from the distance D and overlap O. For example, the waypoint baseline Bw may be a distance along the contour of the slice that the UAV 102 moves between a location where one image is acquired and a location where the next image is acquired (e.g., taking into account the FOV, the distance D from the camera to the surface, and the percentage of overlap O). For example, based on the FOV and the distance D, the surface area covered by each image may be calculated. Furthermore, to ensure that a specified overlap of the two images is achieved, the rotation of the UAV 102 (e.g., around the Z axis in the example of FIG. 1) may be tracked and controlled when moving from a waypoint where an image is acquired to the next waypoint where the next image is acquired. For example, the gimbal 110 may be controlled to point the camera 106 to the next point of interest to counteract the rotation of the UAV 102 while traversing the contour path.
[0035] The UAV 102 may determine a slice baseline Bs from the distance D and the side lap S. For example, the slice baseline distance Bs may be the distance between a slice and an adjacent slice. For example, based on the distance D from the camera 106 to the surface 136, the area of the surface covered by the FOV can be determined, and then based on a specified percentage of the side lap S, the UAV 102 may determine the spacing between slices 140 (i.e., the slice baseline distance Bs).
[0036] After the slice baseline Bs is determined, the UAV 102 may determine the number of slices 140 to apply to the scan object 111 because the slice baseline Bs specifies the allowable distance between adjacent slices. As one example, the UAV may associate an (X, Y, Z) coordinate system 142 to the scan object 111. Various techniques may be used to determine how to associate the coordinate system 142 to the scan object 111. As one example, the orientation of the coordinate system 142 may be based at least in part on the detected shape of the scan object 111 determined from the SDF model. For example, in the example of FIG. 1, since the scan object 111 is an elongated structure that extends vertically upward, the UAV 102 may associate one of the axes, such as the Z axis in this case, with the longest vertical portion of the scan object 111 (e.g., the major edge 138) and may attempt to associate at least one of the other axes, i.e., the X axis and / or the Y axis, with other detected major edges 144 and / or other prominent structures of the scan object 111. As another alternative, the UAV 102 may fit a bounding ellipse to the scan object and associate one axis of the coordinate system, such as the Z axis in this example, to the major axis of the ellipse and one of the X or Y axes to the minor axis of the ellipse. As yet another alternative, the UAV 102 may fit a bounding rectangle to the scan object and assign one axis of the coordinate system 142 to the major axis of the bounding rectangle and another axis of the coordinate system 142 to the minor axis of the bounding rectangle. As yet another alternative, a user may determine how to associate the coordinate system 142 to the scan object. As yet another alternative, the UAV 102 may employ a machine learning model to determine the optimal assignment of the X, Y, and Z axes to the scan object. For example, the machine learning model may be trained on multiple different structures associated with the coordinate system to train the machine learning model to assign the axes of the coordinate system to various shapes and structures.
[0037] After the coordinate system 142 is associated with the scan object 111, for each axis A of the (X, Y, Z) coordinate system 142, the computing device may sample SDF slices along the selected axis A in successive order (e.g., ascending or descending) in steps of Bs, such that A=floor+n*Bs. After the scan object model is divided into multiple slices 140 along the selected axis A at a distance Bs from each other, for each slice, the UAV 102 may determine a contour of D along that slice. The contour may be represented as a set of 1D paths (e.g., from one waypoint to an adjacent waypoint) in a 2D plane corresponding to a particular slice.
[0038] The UAV 102 may further determine the location of the waypoint based on the waypoint baseline Bw for each contour step from one waypoint along the contour path in the 2D plane of each slice to the next adjacent waypoint. Each waypoint is an image acquisition position where the first camera may be operated to acquire an image of the surface of the scan object 111. Furthermore, for each selected waypoint in the slice, the UAV may select a gaze point on the surface of the scan object 111 as a function of the SDF gradient and the slice plane. For example, the gaze point may be a point on the surface of the scan object 111 to which the focus of the first camera 106 is directed to acquire an image of the surface when the UAV 102 is positioned at the corresponding selected waypoint. The rotation of the gimbal 110 and the rotation of the UAV 102 may be controlled to direct the first camera to the gaze point specified for the current waypoint as the UAV 102 moves from one waypoint to the next.
[0039] In addition, the UAV 102 may order the waypoints within each slice in an optimal manner. As one example, the UAV 102 may perform a traveling salesman problem algorithm between the mean positions of the waypoints to form an ordered set of slices containing an ordered set of contours that contain an ordered set of waypoints. This process may be repeated for each of the other two axes to form an ordered set of slices containing an ordered set of contours that contain an ordered set of waypoints for each of the three axes X, Y, and Z of the coordinate system 142.
[0040] In some examples, the UAV 102 may apply a preferred axis ordering, such as Z, X, Y, that keeps all waypoints for image acquisition in a first selected axis and then prioritizes removing waypoints from the second and third axes based on some proximity to an incremental coverage metric. For example, if a selected axis results in complete coverage of all areas of the scan object desired to be scanned, scanning along the other two axes may not be performed at all. However, if the first selected axis results in only partial coverage, the UAV 102 may select one of the remaining axes that results in coverage of the areas remaining uncovered by the slices of the first selected axis. Following selection of the axis and slice to be traversed, the UAV 102 may select a starting location based on minimizing unnecessary travel distances during scanning, such as by starting at an end of one of the first selected axes. Depending on the configuration of the scan target and slices, the UAV 102 may perform an orbital pattern of continuous orbits of the scan target, a lawnmower pattern, a pole pattern, or a combination of these patterns. The UAV 102 may traverse a selected one of the axes in sequence, contour by contour, such as starting at a first end of the axis and moving to the other end. For example, in the illustrated example, the UAV 102 may start at slice 140(1) and move sequentially up slice by slice, e.g., to 140(2), 140(3), etc., or vice versa, starting at slice 140(14) and moving sequentially down slice by slice, e.g., to 140(13), 140(12), etc.
[0041] Additionally, the UAV 102 may be configured to move at a selected constant speed as it traverses the contour path. For example, the UAV 102 may determine the traversal speed based on the selected distance D, the camera exposure time setting, the current lighting conditions, and the amount of motion blur that is acceptable for the intended use of the captured images. Motion blur may be determined based on the distance (in pixels) that a point on the scanned surface travels on the camera's image sensor between the time that the exposure begins and the time that the exposure ends. For example, a threshold level of acceptable motion blur may be established depending on the desired sharpness of the image. Based on a threshold level of acceptable subject blur, the capabilities of the first camera 106 (e.g., determined based on a pre-empirically determined camera model for the first camera 106 or provided by the camera manufacturer), the current lighting conditions, e.g., determined by a light sensor associated with the first camera 106, and the distance D, the UAV 102 may determine a maximum traversal speed for traversing the contour path of the slice and may traverse the contour path at a constant speed, e.g., based on the maximum traversal speed, while acquiring images at waypoints at regular intervals without having to slow down or stop at each waypoint.
[0042] In some examples, the speed on different sides of the scan target may be different. For example, if one side of the scan target 111 is shaded while the other side is in bright sunlight, the UAV 102 may move at a slower constant speed on the shaded side and a faster constant speed on the sunlit side. Alternatively, as another example, the UAV 102 may determine a maximum constant speed for the portion of the scan target 111 that has the lowest light level and use that speed as the constant image acquisition speed for the entire scan. Many other variations will be apparent to one of ordinary skill in the art having the benefit of the disclosure herein.
[0043] During the scan of the scan object 111, such as when traversing a contour path (not shown in FIG. 1 ) corresponding to slices 140(1)-140(14), the UAV 102 may identify additional contours or other surfaces of the scan object to be scanned. For example, as the UAV 102 is performing the scan, the UAV 102 may perform a replanning of the scan plan in real time. For example, the UAV 102 may keep the originally determined slices, and may join the current contour to the new contour if the current contour and the new contour are approximately on the same plane (e.g., within a difference less than the size of the slice baseline Bs). In another example, if a previously undiscovered surface is discovered, the UAV 102 may perform a survey operation, such as by interrupting the current scan and flying around and further away from the scan object to generate a mesh 3D model of the scan object including the new area, to obtain better visualization and effective time estimation, and determine a new scan plan including the additional slices.
[0044] In some examples, a coverage metric may be employed to ensure that the UAV 102 performs a scan of all desired areas of the scan object 111. As one example, an initial coverage mesh may be created from multiple contour path waypoints of multiple slices. The UAV 102 may determine incomplete areas and may fill these areas by acquiring images based on a comparison of the initial coverage mesh with a 3D model determined and updated during the scan based on integrating additional point locations into the 3D model by acquiring images using the second camera 108 during the scan. For each incomplete area identified based on the comparison, the UAV 102 may determine a surface normal at that location and may travel to that location to acquire one or more images of the surface at a distance D from the surface.
[0045] In some examples, the gaze point may be determined based on the gradient of the SDF model. If a Z slice is located near the ground, only the gradient of the SDF in the slice plane may be used, thereby keeping the gimbal 110 level for that slice and preventing a situation where the gradient would otherwise cause the camera to point toward the ground. Additionally, for slices that may be traversed using a lawnmower pattern, the gimbal 110 may be controlled such that the camera 106 is tilted along the contour to capture a tilted surface that is small relative to the distance as the UAV 102 moves in the opposite direction for each successive slice to traverse the rows of the lawnmower pattern back and forth. A similar effect may be achieved for the orbital pattern by reversing the direction of the orbit with alternating slices that include a gaze point skewed forward.
[0046] Based on the scan plan determined for traversing the slices, the UAV 102 may traverse the contour slice by slice in an orderly manner as described above, possibly at a constant rate acquiring images at a constant rate. Furthermore, in some examples, while the UAV 102 is acquiring images using the first camera 106, the UAV 102 may use images from the second camera 108 to update the 3D model and improve the accuracy of the 3D model. The UAV 102 may autonomously navigate and acquire images of the scan target based on the scan plan. For example, based on the scan plan, gapless images may be acquired while the UAV 102 flies continuously from one contour to the next while avoiding collisions with obstacles that may be present.
[0047] During the scan of the scan target based on the scan plan, the UAV 102 may use 3D reconstruction techniques to generate a higher resolution version of the 3D model of the scan target in real time based at least in part on newly detected distances to points on the surface, etc. In the examples herein, "real time" may include "near real time" and may refer to performing the mentioned processing or other operations without undue delay, for example, while the UAV 102 is still flying after acquisition of one or more images and / or when processing power becomes available while the UAV 102 is performing a scan, moving between waypoints, etc. The actual amount of time for real time processing herein may vary based on the processing power of the on-board processor and other components of the UAV 102. For example, the UAV 102 herein may perform any of several different processing operations in real time, such as updating the 3D model, updating the scan plan, etc.
[0048] In some examples, the scanning and 3D reconstruction may be performed iteratively, such as by adding further points or other information representative of the surface of the scan object 111 to the 3D model during the scanning of the scan object 111 according to the initial scan plan, and dynamically updating the initial scan plan based on the information added to the 3D model. Thus, the quality of the surface information of the 3D model may continue to be improved as the UAV 102 navigates the scan plan and additional surfaces may be discovered. In some cases, the scan plan may be dynamically updated to cover new points not previously included, or to avoid obstacles not previously identified, etc., as the accuracy, coverage, etc. of the 3D model are iteratively improved.
[0049] For example, after the UAV 102 has completed a scan or at least a portion of a scan of the object, higher resolution 3D models and images acquired during the scan may be exported from the UAV 102 to the service computing device 126, such as via the network 113. In some examples, as the scan of the object 111 is being performed, acquired images and / or 3D model information may be wirelessly transmitted to the controller 104 and / or the service computing device 126. In some cases, the images may be correlated with points on the 3D model to enable creation of a textured 3D model of the object, such as for viewing the object 111, for high resolution inspection of the object 111, or for performing any of a variety of other observation, computer graphic, or computer modeling operations that may be performed using such high resolution 3D model and high resolution images.
[0050] 2 illustrates an example 200 of determining a contour path and waypoints for a slice of a scan object 202 according to some implementations. In this example, it is assumed that the scan object 202 has a star-shaped cross section, as shown in the figure. It is further assumed that the cross section depicts an XY plane, with the Z axis pointing up, as shown by coordinate system 204, and represents a slice 206 taken across the Z axis similar to slice 140 described above with respect to FIG. 1. Of course, in other examples, slice 206 may be taken across the X or Y axis.
[0051] In some cases, each slice may be used to determine a sampled contour along a 2D plane of the SDF model at the location of the slice, such as by keeping one axis of the coordinate system constant. For example, a 2D contour may be determined at the zero contour, i.e., at a desired distance from the surface of the scanned object. Slices may be acquired in any plane, or may be aligned to two of the axes by keeping the third axis constant to simplify calculations. For example, keeping Z=1 allows a cross section of Z in the XY plane to be obtained. Multiple slices may be determined by acquiring slices at specified intervals (e.g., based on the slice baseline distance Bs). For each slice, a contour at a distance D from the surface may be sampled to obtain a 1D contour path. Then, waypoints may be sampled along the contour path at intervals related to the specified overlap, i.e., the waypoint baseline distance Bw. The UAV 102 may move along the contour path while acquiring images at regular intervals, typically looking inward along a signed distance normal or some derivative of the contour shape. In some cases, the traverse pattern, eg, lawnmower, track, pole, etc., may correspond to a particular instance of the scene shape and selected slice size.
[0052] As previously discussed, the UAV 102 may determine a distance D from the surface 207 of the scan target 202 to use as a baseline distance between the first camera 106 and the surface 207 when acquiring images for the scan. In some cases, the distance D may be a predetermined default value designated for the particular type of scan to be performed based on a desired level of detail of the images acquired, an intended use of the scan results, etc. Additionally or alternatively, the distance D may be specified by the user 112 when the user instructs the UAV 102 to perform a scan of the specified scan target 202. Similarly, the user 112 may specify an overlap O and side lap S for the scan, and / or O and S may be default values for the particular type of scan, the particular type of scan target, etc.
[0053] The UAV 102 may determine a contour path 208 for the UAV 102 to traverse relative to the surface 207 based on the distance D. Additionally, the UAV 102 may determine a waypoint baseline Bw based on the distance D, the desired overlap O, and the FOV of the first camera 106 used to acquire images for the scan. For example, the waypoint baseline Bw may be a distance along the contour path 208 of the slice that the UAV 102 travels between a location where a first image is acquired at, for example, the first waypoint 210(1) and a location when a next image is acquired at, for example, the second waypoint 210(2). For example, based on the FOV and the distance D, a surface area covered by each image acquisition may be calculated. Additionally, by evenly spacing the waypoints according to the waypoint baseline Bw, in some examples, the UAV 102 may be configured to traverse the contour path 208 at a constant speed while acquiring images of the surface 207 at regular intervals. This arrangement can optimize the operation of the UAV 102 when acquiring images while also ensuring complete and orderly coverage of the scan target 202.
[0054] Further, the positions of the gaze points 212 on the surface 207 of the scan target 202 may be determined. Each gaze point 212 may correspond to one of the waypoints 210 and may be the focal point of the first camera 106 on the surface 207 of the scan target 202 when acquiring an image at the corresponding waypoint 210. For example, the first gaze point 212(1) may correspond to the first waypoint 210(1), the second gaze point 212(2) may correspond to the second waypoint 210(2), the third gaze point 212(3) may correspond to the third waypoint 210(3), etc. In some cases, the waypoint baseline distance Bw may be used to determine the distance between the gaze points (not shown in FIG. 2). Then, in some cases, the positions of the waypoints 210 may be determined based on the gaze points 212.
[0055] Additionally, to ensure that a specified overlap of the two images is achieved, the rotation of the UAV 102, e.g., about the Z axis, may be tracked and controlled, such as when moving from one waypoint 210 at which an image is acquired to the next waypoint 210 at which the next image is acquired. For example, the gimbal 110 of the first camera 106 may be controlled to ensure that the camera is pointed at the correct gaze point 222 on the surface 207 when the UAV 102 reaches the waypoint 210 corresponding to the correct gaze point 222. For example, the gimbal may be controlled by the UAV 102 to rotate the camera 106 to counteract a rotation of the body of the UAV 102 that may occur while traversing the contour path 208 from one waypoint 210 to the next waypoint 210. Thus, based on the waypoint baseline distance Bw, the UAV 102 may determine multiple spaced waypoints 210 along the contour path 208. As previously described, this process may be repeated for each slice of the scanned object. Following determination of the waypoints, a scan pattern may be determined for efficiently traversing the waypoints.
[0056] 3A and 3B show an example of determining a contour path of a scan object according to some implementations. FIG. 3A shows an example 300 of determining a slice of a scan object 302 according to some implementations. In this example, assume that the UAV 102 has associated a coordinate system 304 with the scan object 302, with the Y axis of the coordinate system 304 aligned with the longest major edge 306 of the scan object 302 and the X axis aligned with another edge 308. Further, the UAV 102 has sliced the scan object 302 across the Y axis to generate a number of slices spaced apart by a slice baseline distance Bs, and has determined a number of corresponding contour paths 310 spaced apart by a distance D from the surface of the scan object 302. Further, assume that the UAV 102 has determined that the contour path 310 results in coverage of all surfaces to be scanned. Thus, the UAV 102 may determine the location of a waypoint for each of the contours 310 and may select a pattern for performing the scan, such as a lawnmower pattern. For example, to scan the scan target 302, a lawnmower pattern may be performed by starting at the left or right end of the contour path 310 and flying back and forth along successively adjacent contour paths 310 until the opposite end is reached.
[0057] 3B illustrates an example 320 of determining slices of a scan object 322 according to some implementations. In this example, the scan object 322 is cubic in shape, and the UAV 102 aligns the coordinate system 322 to the horizontal and vertical edges of the cube. Additionally, the UAV 102 determines a first plurality of slices along the Z axis to determine a corresponding plurality of first contour paths 326 spaced apart from each other by a scan baseline distance Bs, spaced apart from a side 328 of the scan object 322, corresponding to the slices in the Z axis.
[0058] Additionally, the UAV 102 determines a second plurality of slices acquired across the Y-axis to determine a corresponding second plurality of contour paths 330 spaced apart from each other by a scan baseline distance Bs, spaced apart from a top surface 332 of the scan target 322 by a distance D. The UAV 102 may determine a plurality of waypoints for each contour 326, 330. Furthermore, if the waypoints may overlap in coverage of the surface of the scan target 322, the UAV 102 may remove some of those waypoints to prevent overlapping image acquisition on the same surface of the scan target 322. For example, in this example, a Y-axis waypoint for acquiring an image of the side surface 328 of the scan target 322 may be removed from the scan plan performed to scan the scan target 322. Furthermore, when determining a scan plan for scanning the scan target 322, the UAV may perform an orbital pattern to traverse a first plurality of contour paths 326 to scan a side surface 328 of the scan target 322, such as by starting from the bottom or top of the scan target 322, and then apply a lawnmower pattern to traverse a second plurality of contour paths 330 to scan a top surface 332 of the scan target 322. Additionally, although multiple examples of determining contour paths for various types of scan targets are described herein, many variations will be apparent to one of ordinary skill in the art having the benefit of the disclosure herein.
[0059] 4A and 4B show examples of determining a contour path of a scan object according to some implementations. FIG. 4A shows an example 400 of determining a contour path of a vertical pillar relative to a scan object 402 according to some implementations. In this example, assume that the UAV 102 associates a coordinate system 404 with the scan object 402 and aligns the Z axis of the coordinate system 404 with the longest major edge 406 of the scan object 402. Furthermore, the UAV 102 slices the scan object 402 across the Y axis to determine multiple slices in the XZ plane to obtain multiple vertical pillar contour paths 408, respectively. The contour paths 408 are spaced apart from each other by a distance equal to or less than the slice baseline distance Bs. After the UAV 102 determines the location of the waypoint along the contour path 408, the UAV 102 may traverse the contour path 408 of the vertical pillar by starting at the bottom of a first contour path 408, moving up, then moving back down the opposite side of the contour path 408, moving to the next adjacent contour path 408, and repeating until all contour paths 408 have been traversed.
[0060] FIG. 4B illustrates an example 420 of determining multiple contour paths for a slice of a scan object 422 according to some implementations. In this example, assume that the scan object 422 has a triangular cross-section, as shown in the figure. Further assume that the cross-section represents an XY plane with the Z axis pointing up, as illustrated by the coordinate system 424, and represents a slice 426 taken across the Z axis. In this example, assume that the UAV 102 is instructed to perform three scans of the scan object at three different distances D1, D2, and D3, respectively, from a surface 428 of the scan object 422. Thus, the UAV 102 may determine three contour paths corresponding to the three different distances, namely, a first contour path 430 located at a distance D1 from the surface 428, a second contour path 432 located at a distance D2 from the surface 428, and a third contour path 434 located at a distance D3 from the surface 428. As one example, the UAV 102 may traverse all contour paths at distance D3, then all contour paths at distance D2, and then all contour paths at distance D1. As another example, the UAV 102 may traverse three contour paths at D3, D2, and D1 for each slice 426 before moving up or down the Z axis to the next adjacent slice. In some examples, the speed at which the UAV 102 traverses each contour path 430, 432, and 434 may be different. For example, the traversal speed of the contour path 434 may be faster than the traversal speed of the contour path 432, which may be faster than the traversal speed of the contour path 430, to reduce the amount of subject blur that may occur in an image captured at the contour path 430 closer to the surface 428, to allow for the capture of a more detailed image, etc.
[0061] 5 and 6 include flow diagrams illustrating example processes according to some implementations. These processes are illustrated as a collection of blocks in a logical flow diagram that represent a sequence of operations that may be implemented in part or in whole in hardware, software, or a combination thereof. In the context of software, the blocks may represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, programs the processor to perform the recited operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the blocks are described should not be construed as limiting. Any number of the described blocks can be combined in any order and / or in parallel to implement a process or an alternative process, and not all of the blocks need to be executed. For purposes of illustration, the processes are described with reference to the environments, systems, and devices described in the examples herein, but the processes may be implemented in a wide variety of other environments, systems, and devices.
[0062] 5 is a flow diagram illustrating an example process 500 for scanning a scan target according to some implementations. In some examples, at least a portion of the process 500 may be performed by the UAV 102, such as by executing a scan program and a vehicle control program. Alternatively, in some examples, at least a portion of the process 500 may be performed by a computing device remote from the UAV 102, such as the controller 104 and / or the service computing device 126.
[0063] At 502, the UAV 102 may receive an indication of a scan target. In some examples, the UAV 102 may receive the indication of a scan target from the controller 104, such as based on one or more user inputs made via the controller 104. Alternatively, as another example, the UAV 102 may receive the indication of a scan target from the service computing device 126 or from any other computing device that can communicate with the UAV 102.
[0064] At 504, the UAV 102 may determine a distance D, overlap O, and side lap S to use for the requested scan. In some cases, the distance D, overlap O, and / or side lap S may be predetermined default values designated for the particular type of scan to be performed based on the desired level of detail of the images acquired, the intended use of the scan results, etc. Additionally or alternatively, one or more of these values may be specified by the user 112.
[0065] At 506, the UAV 102 may access a model including a plurality of points in 3D space representing the location of one or more surfaces of the indicated scan target. As one example, the UAV 102 may generate the 3D SDF model based on acquiring a plurality of initial images of the scan target in response to a received indication of the scan target, as described further below with respect to FIG. 6. In other examples, the UAV 102 may receive an existing 3D model of the scan target from the controller 104, the service computing device 126, or another remote computing device.
[0066] At 508, the UAV 102 may determine a waypoint baseline distance Bw and a slice baseline distance Bs for the scan. For example, the UAV 102 may determine the waypoint baseline distance Bw based on the FOV, distance D, and overlap O of the first camera 106. The waypoint baseline Bw may be a distance along the contour of the slice that the UAV 102 moves between the location where one image is acquired and the location where the next image is acquired while maintaining a desired overlap. Similarly, the UAV may determine a slice baseline distance Bs based on the FOV, distance D, and side lap S of the first camera 106. For example, the slice baseline distance Bs may be a distance between a slice and an adjacent slice.
[0067] At 510, the UAV 102 may associate a coordinate system with the scan object. As one example, the UAV 102 may associate one of the axes of the coordinate system with the longest edge or element of the scan object, and may attempt to associate other axes with other portions of the scan object, such as other major edges.
[0068] At 512, for each axis A of the (X, Y, Z) coordinate system, the UAV 102 may determine one or more slices that transversely intersect axis A and are spaced apart on axis A by a slice baseline distance Bs. For example, if an operation is being performed on a 3D SDF model, slices may be determined for the SDF model along each of the three axes of the coordinate system.
[0069] At 514, the UAV 102 may determine, for each slice, a contour of the distance D. As one example, the contour may be determined as a set of multiple 1D paths in the 2D plane of each slice. The multiple 1D paths may together comprise a contour path in 3D space located at the distance D from the surface of the scanned object.
[0070] At 516, for each contour, the UAV 102 may follow the contour and determine the locations of waypoints spaced apart by a distance less than or equal to the waypoint baseline distance Bw.
[0071] At 518, for each waypoint, the UAV 102 may determine a point of regard at which the camera will focus when the UAV 102 is at that respective waypoint. As one example, the point of regard may be determined for each waypoint as a function of the SDF gradient of the slice plane.
[0072] At 520, the UAV 102 may order the contours in each slice. For example, the UAV 102 may use a traveling salesman problem (TSP) algorithm to order the contours based on their respective average positions. Blocks 512-520 may be repeated for each axis of the coordinate system to obtain, for each axis of the coordinate system, an ordered set of slices that contain an ordered set of contours that contain an ordered set of waypoints.
[0073] At 522, the UAV 102 may determine a scan plan for performing the scan. As one example, the UAV 102 may use a preferred axis ordering, such as selecting the Z axis, then the X axis, and then the Y axis. Alternatively, as another example, the UAV 102 may select the axis that contains the most slices. The UAV 102 may then remove slices and / or waypoints from the other two axes based on a proximity metric, an incremental coverage metric, or the like. For example, if an area of the surface to be scanned is already covered by a waypoint in the selected axis, the waypoints in the other two axes for that area may be removed or otherwise unused.
[0074] At 524, the UAV 102 may determine an efficient pattern for executing the scan plan. For example, if the contour path encompasses the scan target, the UAV 102 may first attempt to apply an orbital pattern to traverse the contour path. Alternatively, for a contour path for which an orbital pattern cannot be applied, the UAV 102 may attempt to apply a lawnmower pattern or other efficient pattern that allows movement from one contour path to an adjacent contour path for an adjacent slice following traversal of the first contour path.
[0075] At 526, the UAV 102 may determine a speed at which to traverse each contour path in the scan plan. As one example, the UAV 102 may be configured to traverse at least a portion of the contour path at a constant speed to acquire images of the surface to be scanned at regular intervals in time and / or space. For example, the UAV 102 may determine the traversal speed based on the selected distance D, the camera exposure time setting, the current lighting conditions at the surface to be scanned, and the amount of blur that is acceptable for the intended use of the acquired images. In some cases, a threshold level of acceptable blur may be established depending on the desired sharpness of the image. Based on the threshold level of acceptable blur, the capabilities of the first camera 106, e.g., as determined by an optical sensor associated with the first camera 106, the current lighting conditions, and the distance D, the UAV 102 may determine a constant traversal speed for traversing at least a portion of the contour path in the scan plan.
[0076] In some examples, the speed in different portions of the scan plan may be different. For example, if one portion of the scan target is shaded while another portion is brightly lit, the UAV 102 may move at a slower constant speed in the shaded portion and a faster constant speed for the brightly lit portion. Alternatively, as another example, the UAV 102 may determine a maximum constant speed for the portion of the scan target having the lowest light level and use that speed as the constant image acquisition speed for the entire scan plan. Many other variations will be apparent to one of ordinary skill in the art having the benefit of the disclosure herein.
[0077] At 528, the UAV 102 may execute the scan plan by moving at the determined constant speed as far as possible and acquiring images at regular intervals in time and / or space made possible at least in part by the continuous, structured nature of the contour path.
[0078] 6 is a flow diagram illustrating an example process 600 for generating a 3D model of a scan target according to some implementations. In some examples, at least a portion of the process 600 may be performed by the UAV 102, such as by executing a scan program and a vehicle control program, as described further below. Alternatively, in some examples, at least a portion of the process 600 may be performed by a computing device remote from the UAV 102, such as the controller 104 and / or the service computing device 126.
[0079] At 602, the UAV 102 may receive an indication of a scan target. For example, the UAV 102 may receive the indication of a scan target from the controller 104, such as based on one or more user inputs made via the controller 104. Alternatively, as another example, the UAV 102 may receive the indication of a scan target from the service computing device 126 or from any other computing device that can communicate with the UAV 102.
[0080] At 604, the UAV 102 may determine one or more poses for acquiring images of a surface corresponding to the indicated scan target. For example, if a bounding area or bounding volume is used to specify the scan target, the UAV 102 may determine one or more positions and fields of view for acquiring images of the indicated scan target based on correlating the bounding area or bounding volume with real-world positions to acquire images of any surfaces within the bounding area or bounding volume.
[0081] At 606, the UAV 102 may control the propulsion mechanism 116 of the UAV 102 to fly to the determined one or more attitudes to acquire images at each attitude. As one example, the UAV may acquire images simultaneously with two or more of the second cameras 108, allowing the second cameras 108 to act as a stereoscopic range finder and determine distances to various points on the surface of the scanned object.
[0082] At 608, the UAV 102 may determine a distance from the UAV to the surface captured in the image, such as based on determining disparity between points in associated images. For example, the UAV 102 may employ multi-view stereo analysis of multiple images to determine the respective distances of each point on the scanned surface.
[0083] At 610, the UAV 102 may use the images from the camera 108 to generate a 3D SDF model that includes the locations of multiple points in 3D space. The 3D model may initially be a coarse, low-resolution 3D model and may be improved in accuracy as additional images of the scanned object are acquired by the UAV 102. For example, the lower-resolution 3D SDF model may include multiple points, and the locations of those points in 3D space may be determined based on image analysis, such as determining disparity or other differences between two or more images of the points acquired using multiple image sensors on the UAV 102.
[0084] At 612, the UAV 102 may transmit information regarding the lower resolution 3D model to a remote computing device. For example, model information regarding the 3D SDF model may be transmitted to the controller 104, the service computing device 126, or other computing device remote from the UAV 102 that may be in communication with the UAV 102. Further, as previously mentioned, the 3D SDF model may be used to determine a scan plan for scanning the object as described above with respect to FIG. 5. In addition, although a signed distance function has been described as one technique for determining the location in 3D space of a point on the surface of the object, other techniques will be apparent to one of ordinary skill in the art having the benefit of the disclosure herein.
[0085] The exemplary processes described herein are merely examples of processes provided for illustrative purposes. Many other variations will be apparent to those skilled in the art in light of the disclosure herein. Furthermore, although the disclosure herein sets forth several examples of suitable frameworks, architectures, and environments for carrying out the processes, the implementations herein are not limited to the specific examples shown and described. Furthermore, the disclosure provides various exemplary implementations as described and illustrated in the drawings. However, the disclosure is not limited to the implementations described and illustrated herein, and may extend to other implementations as known or to become known to those skilled in the art.
[0086] 7 illustrates an example configuration of the upper side of a UAV 102 according to some implementations. In this example, the UAV 102 includes a propulsion system 116 that includes four motors 704 and a propeller 706. In this example, the UAV 102 is illustrated as a quadcopter drone, although implementations herein are not limited to such quadcopter drones.
[0087] The UAV 102 includes a number of second cameras 108 mounted on the body 114 of the UAV 102, which may possibly be used as navigation cameras. The UAV 102 further includes an aimable first camera 106, which may include a higher resolution image sensor than the image sensor of the wider angle camera 108. In some cases, the first camera 106 includes a fixed focal length lens. In other cases, the first camera 106 may include a mechanically controllable optical zoomable lens. The first camera 106 is mounted on a gimbal 110, which allows aiming of the first camera 106 within a hemispherical area of about 180 degrees to support stable, low-shake image acquisition and object tracking. For example, the first camera 106 may be used to acquire high-resolution images of a target object, provide object tracking video, or for various other operations.
[0088] In this example, three second cameras 108 are spaced apart on the top side 708 of the UAV 102 and covered by fisheye lenses that provide a wide field of view and support stereoscopic computer vision. The wider-angle cameras 108 on the top side 708 of the UAV 102, as well as the wider-angle cameras 108 on the bottom side described below, can be precisely calibrated to each other after mounting to the body 114 of the UAV 102. As a result of the calibration, for each pixel of the image acquired by each wider-angle camera, a precise corresponding three-dimensional (3D) orientation with respect to a virtual sphere surrounding the UAV can be predetermined. In some cases, six wider-angle cameras 108 with a sufficiently wide FOV (e.g., 180-degree FOV, 200-degree FOV, etc.) are employed and positioned on the body 114 of the UAV 102 to cover the entire spherical space around the UAV 102.
[0089] FIG. 8 illustrates an example configuration of the bottom side of the UAV 102 according to some implementations. In this perspective view, three additional second cameras 108 are shown positioned on the bottom side 802 of the UAV 102. The second cameras 108 on the bottom side 802 may also be covered by respective fisheye lenses that provide a wide field of view and support stereoscopic computer vision. This array of second cameras 108 (e.g., three on the top side and three on the bottom side of the UAV 102) may enable visual inertial odometry (VIO) for high-resolution localization and obstacle detection and avoidance. For example, the array of second cameras 108 may be used to provide image information that may be used to scan the surrounding area to obtain range data and generate a range map showing distances to objects detected within the FOV of the second cameras 108 for use during autonomous navigation of the UAV 102 or to determine the distance of a surface from the UAV 102, etc.
[0090] The UAV 102 may include a battery pack 810 attached to the bottom side 802 of the UAV 102 with conductive contacts 812 that allow charging of the battery. The UAV 102 also includes various other electronic and mechanical components, in addition to an internal processing unit including one or more processors and computer-readable media (not shown in FIG. 8). For example, the UAV 102 may include a hardware configuration as described below with respect to FIG. 6.
[0091] 9 illustrates selected components of an example UAV 102 according to some implementations. In the examples herein, the UAV 102 may be referred to as a "drone" and may be implemented as any type of UAV capable of controlled flight without a human pilot on board. For example, the UAV 102 may be autonomously controlled by one or more on-board processors 902 executing one or more executable programs. Additionally or alternatively, the UAV 102 may be controlled via a remote controller, such as via a remotely located controller 104 operated by a human pilot and / or controlled by executable programs executing on or in conjunction with the controller 104.
[0092] In the illustrated example, the UAV 102 includes one or more processors 902 and one or more computer-readable media 904. For example, the one or more 902 may execute software, executable instructions, etc. to control flight, navigation, image acquisition, and other functions of the UAV 102. Each processor 902 may be a single processing unit or multiple processing units and may include single or multiple computing units or multiple processing cores. The processor 902 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, graphics processing units, state machines, logic circuits, and / or any device that manipulates signals based on operational instructions. For example, the processor 902 may be one or more hardware processors and / or logic circuits of any suitable type that are specifically programmed or configured to execute the algorithms and processes described herein. The processor 902 may be configured to fetch and execute computer-readable instructions stored on the computer-readable media 904 that can program the processor 902 to perform the functions described herein.
[0093] The computer readable medium 904 may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of technology for storage of information such as computer readable instructions, data structures, program modules, or other executable code and data. Such computer readable medium 904 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage, solid-state storage, magnetic storage, or any other medium that can be used to store the desired information and that can be accessed by a computing device. Depending on the configuration, the computer readable medium 904 may be a type of computer readable medium and / or a tangible non-transitory medium, insofar as non-transitory computer readable medium as referred to herein excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.
[0094] The computer-readable medium 904 may be used to store any number of functional elements executable by the processor 902. In many implementations, these functional elements include instructions or programs executable by the processor 902 that, when executed, specifically configure the one or more processors 902 to perform at least some of the actions attributed to the UAV 102 described above. The functional elements stored on the computer-readable medium 904 may include a vehicle control program 906 that may be executed to point the first camera 106 at an object and control the first camera 106 to acquire an image of the object, in addition to controlling the autonomous navigation of the UAV 102. The functional elements further include a scan program 908 that may be executed by the one or more processors to perform at least some of the scan functions of the scan object described herein. In addition, the functional elements may include a web application 909 that may be accessed by the controller 104 for use in controlling the UAV 102. For example, the web application 909 may run in a browser on a user device connected to the controller 104 to enable a user to receive images of the current viewpoint of the UAV 102, receive scanned images, 3D models, etc., while viewing the scanning progress of the UAV 102 while scanning the target.
[0095] In addition, the computer-readable medium 904 may store data used to perform the navigation and scanning operations described herein. Thus, the computer-readable medium 404 may store, at least temporarily, acquired images 910, sensor data 912, one or more 3D models 914, and one or more scan plans 916. In addition, the computer-readable medium may store navigation / tracking information 918 that may be used to navigate the UAV 102 according to one or more instructions and to provide information related to one or more objects for imaging. In addition, the UAV 102 may include many other logical, program, and physical components, and the logical, program, and physical components described herein are merely examples relevant to this description.
[0096] To aid in navigation, the UAV 102 may include a Global Navigation Satellite System (GNSS) receiver or other satellite positioning system receiver 920 on board the UAV 102. The GNSS receiver 920 may be capable of receiving signals from one or more satellites of a GNSS, such as the Global Navigation Satellite (GPS) system, the Russian Global Navigation Satellite System (GLONASS), the Chinese Beidou Navigation System (BDS), the European Union's Galileo system, the Japanese Quasi-Zenith Satellite System (QZSS), the Indian Regional Navigation Satellite System (IRNSS), and the like.
[0097] The UAV 102 may further include an inertial measurement unit (IMU) 922. In some embodiments, the IMU 922 may be configured to detect linear acceleration and gravity using one or more accelerometers and to detect rotational rate using one or more gyroscopes. As one example, the IMU 922 may be self-contained having a three-axis gyroscope, a three-axis accelerometer, and an embedded processor for processing inputs from the gyroscope and accelerometer to provide outputs such as acceleration, attitude, and the like. For example, the IMU 922 may measure and report velocity, acceleration, orientation, and gravity at the UAV 102, such as by using a combination of a gyroscope and an accelerometer. Additionally, the UAV 102 may include other sensors 924, such as a magnetometer, a barometer, a proximity sensor, a lidar, a radar, an ultrasonic, or any of a variety of other types of sensors known in the art.
[0098] Additionally, the UAV 102 may include one or more communication interfaces 926, one or more flight controllers 928, one or more propulsion devices 116, and an image acquisition system 930. The image acquisition system 930 may include the second camera 108, the first camera 106, and one or more stabilization and tracking devices 934, such as the gimbal 110 described above.
[0099] Additionally, the UAV 102 may include an image transmission system 936, input / output (I / O) devices 938, and a power system 940. The components included in the UAV 102 may be capable of communicating with at least one or more processors 902 via one or more communication buses, signal lines (not shown), or the like.
[0100] The UAV 102 may include more or fewer components than those shown in the example of Figure 9, may combine two or more components as a functional unit, or may have a different configuration or arrangement of components. Some of the various components of the example UAV 102 shown in Figure 9 may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application specific integrated circuits.
[0101] The flight controller 928 may include a combination of hardware and / or software configured to receive input data (e.g., sensor data, image data, generated trajectories, or other instructions) from the vehicle control program 906, interpret the data and / or instructions, and output control signals to the propulsion device 116 of the UAV 102. Alternatively, or in addition, the flight controller 928 may be configured to receive control commands generated by another component or device (e.g., the processor 902 and / or the controller 104), interpret those control commands, and generate control signals to the propulsion device 116 of the UAV 102. In some implementations, the aforementioned vehicle control program 906 of the UAV 102 may include any one or more of the flight controller 928 and / or other components of the UAV 102. Alternatively, the flight controller 928 may exist as a separate component from the vehicle control program 906.
[0102] The communication interface 926 may enable the transmission and reception of communication signals, for example, via a radio frequency (RF) transceiver. In some implementations, the communication interface 926 may include RF circuitry (not shown in FIG. 9). In such implementations, the RF circuitry may convert electrical signals to / from electromagnetic signals and communicate with communication networks and other communication devices via electromagnetic signals. The RF circuitry may include known circuitry for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and the like. The RF circuitry may facilitate the transmission and reception of data over a communication network (including public, private, local, and wide area). For example, communication may be over a network of networks, such as a wide area network (WAN), a local area network (LAN), or the Internet.
[0103] The communication interface 926 may include one or more interfaces and hardware components for enabling communication with various other devices, via one or more networks, etc. For example, the communication interface 926 may enable communication via one or more of the Internet, cable networks, cellular networks, wireless networks (e.g., Wi-Fi) and wired networks (e.g., fiber optics and Ethernet), and short-range wireless communications such as BLUETOOTH, etc., as further enumerated elsewhere herein. For example, 900 MHz, 2.4 GHz, and 5.8 GHz are the most common radio frequencies used for communication with UAVs, although implementations herein are not limited to any particular frequency.
[0104] The input / output (I / O) devices 938 may include physical buttons (e.g., push buttons, rocker buttons, etc.), LEDs, dials, displays, touch screen displays, speakers, etc. that may be used to interact with or otherwise operate certain functions of the UAV 102. The UAV 102 also includes a power system 940 for providing power to the various components. The power system 940 may include a power management system, one or more power sources (e.g., batteries, alternating current, etc.), a charging system, power failure detection circuitry, power conversion devices or converters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in a computerized device.
[0105] In some examples, similar to an aircraft, the UAV 102 may utilize fixed wings or other aerodynamic surfaces along with one or more propulsion devices 116 to achieve lift and navigation. Alternatively, in other examples, similar to a helicopter, the UAV 102 may directly use one or more propulsion devices 116 to achieve lift and navigation against gravity. Lift through propulsion (as in a helicopter) may provide advantages in some implementations because it allows for more controlled movement along all axes compared to UAVs that use fixed aerodynamic surfaces for lift.
[0106] The UAV 102 shown in Figures 1 and 7-9 is an example provided for illustrative purposes. A UAV 102 according to the present disclosure may include more or less components than those shown. For example, while a quadcopter is shown, the UAV 102 is not limited to any particular UAV configuration and may include a hexacopter, an octocopter, a fixed-wing aircraft, or any other type of independently pilotable aircraft, as would be apparent to one of ordinary skill in the art having the benefit of the disclosure herein. Additionally, although techniques are described herein for controlling the navigation of an autonomous UAV 102 to perform a scan of a scan target, the described techniques may be similarly applied to guided navigation by other types of vehicles (e.g., spacecraft, land vehicles, marine vehicles, underwater vehicles, etc.).
[0107] FIG. 10 illustrates an example configuration of a controller 104 including a computing device 1001 according to some implementations. The controller 104 may present a graphical user interface (GUI) 1002 for controlling the UAV 102 and viewing images 118 received from the UAV 102. The controller 104 may include a touch screen 1004, which may correspond to the display 124 described above with respect to FIG. 1. The touch screen 1004 may provide virtual controls and status indicators for controlling and viewing the status of the UAV 102. For example, a camera settings virtual control 1008 may allow a user to control the resolution and other settings of at least the first camera 106 on the UAV 102. Additionally, a battery charge level indicator 1010 may indicate the current status of the battery on the UAV 102. A signal strength indicator 1012 may indicate the current signal strength of a communication signal with the UAV 102. A settings virtual control 1014 may allow a user to control the settings of the controller 104. Additionally, a map virtual control 1016 may allow a user to view the location of the UAV 102 on a map. A home virtual control 1018 may allow a user to return to a home screen of the user interface 1002. A record virtual control 1020 may allow a user to control starting or stopping the recording of the scene currently in the field of view of the first camera 106. A skill virtual control 1022 may allow a user to control the skill settings of the UAV 102. A manual virtual control 1024 may allow a user to switch between manually piloting the UAV 102 or allowing the UAV 102 to pilot itself autonomously.
[0108] In addition, the user interface may present an image 1026 (e.g., a live video image) of the current field of view of the UAV 102. In this example, the touch screen 1004 is part of a computing device 1001, such as a smartphone, tablet computing device, or other computing device that may be attached to the controller 104 using a controller accessory 1034. The controller 104 may further include a controller body 1036 that includes multiple physical controls that may be used to manually control the UAV 102, such as a left joystick 1038, a right joystick 1040, a home button 1042, a launch / land button 1044, an LED status indicator 1046 that indicates the status of the controller 104, and other physical controls not visible in this view. In some examples, a Wi-Fi antenna may be included in the controller accessory 1034 so that the controller 104 can provide a communication range extension capability for communication with the UAV 102 at distances longer than may be possible with the computing device 1001 alone.
[0109] In some cases, the computing device 1001 (or another computing device remote from the UAV 102) may execute an application on a processor of the computing device 1001. As one example, the application may include a browser running a web application that may be provided or otherwise provided to the computing device 1001 by the UAV 102. For example, the web application (or another application running on the computing device 1001) may provide the user interface 1002 described above, and may provide other functionality described herein with respect to the computing device 1001, such as enabling communication with the UAV 102, enabling remote control of the UAV 102, etc. Additionally, in some cases, the application may enable wireless connection of the computing device 1001 to the controller 104, such as via BLUETOOTH radio, Wi-Fi, etc.
[0110] In some implementations, some of the processing that might normally be performed by the UAV 102 (e.g., image processing and control functions) may instead be performed by an application executing on a processor of the computing device 1001 remote from the UAV 102. Additionally, in some examples, the processing load may be split between a processor on the UAV 102 and a processor on the computing device 1001, such as to achieve reduced processing time. Many other variations will be apparent to those of skill in the art having the benefit of the disclosure herein.
[0111] In the illustrated example, it is assumed that a user has selected the scan target 111 using the controller 104 to indicate to the UAV 102 that the scan target 111 should be scanned as described above with respect to FIG. 1. Thus, the user may select an acquired image of the desired scan target 111 and may manipulate the user interface 1002 to draw a polygon or other 2D shape 1050 around the scan target or a portion of the scan target 111 to designate the scan target 111 to the UAV 102. In this manner, the user may designate an area in the user interface 1002 to indicate the scan target 111 to the UAV 102. For example, the user may control at least three reference points, i.e., handles 1054, such as with fingers 1052, to draw a polygonal perimeter (i.e., boundary) around the scan target 111. The UAV 102 may be configured to survey the scan target 111 within the designated perimeter and, in some examples, may present a prism volume around the detected scan target corresponding to the designated perimeter. For example, a user may be able to adjust the distance to the surface and other scan parameters before the UAV 102 begins scanning.
[0112] Additionally, while a polygon is shown in the example of FIG. 8, in other examples, any other technique may be used to create a perimeter around or on the scan object. For example, the user may draw a circle, an oval, an irregular line, etc. on or around the scan object in the user interface 1002. Alternatively, in other examples, the user interface 1002 may allow manipulation of a bounding volume rather than a bounding area to indicate the scan object 111 to the UAV 102. In yet other examples, the user may tap or otherwise select an image of the scan object 111 in the user interface 1002 to indicate the scan object to the UAV 102. In some examples, the UAV 102 may employ a machine learning model to recognize the scan object 111 based on an indication received via the user interface 1002. As another example, the user may provide the UAV 102 with specified latitude and longitude coordinates of the scan object 111, and the UAV 102 may navigate itself to the specified coordinates to acquire an image of the scan object 111. Many other variations will be apparent to those skilled in the art having the benefit of this disclosure.
[0113] 11 illustrates select components of an example controller 104 according to some implementations. The controller 104, in this example, may include components such as at least one processor 1102, one or more computer-readable media 1104, one or more communication interfaces 1106, and one or more input / output (I / O) devices 1108. For example, at least a portion of the processor 1102, computer-readable media, communication interfaces 1106, and I / O devices 1108 may be provided by a computing device 1001 (not shown separately in FIG. 11 ) that is connected to or otherwise included with the controller 104.
[0114] Each processor 1102 may itself comprise one or more processors or processing cores. For example, the processor 1102 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operational instructions. In some cases, the processor 1102 may be one or more hardware processors and / or logic circuits of any suitable type that are specifically programmed or configured to execute the algorithms and processes described herein. The processor 1102 may be configured to fetch and execute computer-readable processor-executable instructions stored on the computer-readable medium 1104.
[0115] Depending on the configuration of the controller 104, the computer-readable medium 1104 may be an example of a tangible non-transitory computer storage medium and may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of technology for storage of information such as computer-readable processor-executable instructions, data structures, program modules, or other data. The computer-readable medium 1104 may include, but is not limited to, RAM, ROM, EEPROM, flash memory, solid-state storage, magnetic disk storage, optical storage, and / or other computer-readable media technologies. Furthermore, in some cases, the controller 104 may access external storage, such as a storage system, storage array, network-attached storage, storage area network, cloud storage, or any other medium that may be used to store information and that may be accessed by the processor 1102 directly or via another computing device or network. Thus, the computer-readable medium 1104 may be a computer storage medium that may store instructions, modules, or components that may be executed by the processor 1102. Additionally, non-transitory computer readable media, when referred to, excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.
[0116] The computer-readable medium 1104 may be used to store and maintain any number of functional elements executable by the processor 1102. In some implementations, these functional elements include instructions or programs executable by the processor 1102 that, when executed, implement operational logic for performing the actions and services attributed to the controller 104 described above. The functional elements of the controller 104 stored on the computer-readable medium 1104 may include a user application 1108 that may enable a user to remotely control the UAV 102 using the controller, as previously described. In some examples, the user application 1108 may access a web application on the UAV 102, while in other examples, the user application 1108 may be a standalone controller application.
[0117] In addition, the computer readable medium 1104 may store data, data structures, etc. used by the functional elements. For example, the computer readable medium 1104 may at least temporarily store images 118 received from the UAV 102, and in some examples may store 3D model information 120 received from the UAV 102. Depending on the type of controller 104, the computer readable medium 1104 may optionally include other functional elements and data, such as applications, programs, drivers, etc., and other programs and data 1110 that may include data used or generated by the functional elements. Additionally, the controller 104 may include many other logical, program, and physical components, and the logical, program, and physical components described are merely examples relevant to the description herein.
[0118] Communications interface 1106 may include one or more interfaces and hardware components for enabling communication with various other devices, such as via network 113 or directly. For example, communications interface 1106 may enable communication via one or more of the Internet, cable networks, cellular networks, wireless networks (e.g., Wi-Fi) and wired networks, one-way or two-way radio transmissions, and short-range communications such as BLUETOOTH®, etc., as further described elsewhere herein.
[0119] In addition, the controller 104 may include a display 124, which may include a touch screen 1004 as described above with respect to FIGURE 10. The controller 104 may further include a number of other I / O devices 1112 as described above with respect to FIGURE 10.
[0120] Clause 1. An unmanned aerial vehicle (UAV), comprising: a first camera mounted on the UAV; and one or more processors configured to perform operations according to executable instructions, the operations including: determining a surface to be scanned; determining a distance from the surface to be scanned as a selected distance associated with acquiring an image; determining a maximum speed for traversing at least the first portion of the surface based at least on illumination associated with at least the first portion of the surface and based on the selected distance; and navigating the UAV at a speed based at least on the determined maximum speed relative to at least the first portion of the surface to be scanned based on the selected distance while acquiring an image of the at least the first portion of the surface to be scanned.
[0121] Clause 2. The UAV of clause 1, wherein the operations further include acquiring an image of a first portion of the scanned surface while flying at or below the maximum speed, determining that illumination associated with a second portion of the scanned surface is different from illumination associated with the first portion of the surface, and determining a maximum speed for traversing the second portion of the surface based at least on the illumination associated with the second portion of the surface and based on the selected distance, the maximum speed for traversing the second portion of the surface being different from the maximum speed determined for traversing the first portion of the surface.
[0122] Clause 3. The UAV of clause 1, wherein the operation of determining the maximum speed is further based on a threshold level of subject blur determined to be acceptable for the captured image.
[0123] Clause 4. The UAV of clause 1, wherein the operations further include determining a plurality of contour paths spaced apart from one another along at least one axis associated with the scanned object, each contour path being spaced apart from a surface of the scanned object based on a selected distance, and determining a plurality of image acquisition locations for each contour path, each image acquisition location indicating a respective location where a respective image of the surface of the scanned object should be acquired, and navigating the UAV at a speed based on the determined maximum speed relative to at least a first portion of the surface of the scanned object based on the selected distance while acquiring images of at least a first portion of the surface of the scanned object includes navigating the UAV along one or more of the contour paths at the determined speed while acquiring images of the surface of the scanned object based on the image acquisition locations.
[0124] Clause 5. The UAV of clause 5, wherein the operations further include associating a coordinate system with the scan object based at least on a configuration of the scan object, the coordinate system including at least one axis, and determining a plurality of spaced contour paths along the at least one axis based at least on respective contours of a surface of the scan object.
[0125] Clause 6. The UAV of clause 1, wherein the operations further include receiving from the computing device an indication of the scanned object, causing the UAV to assume one or more positions to respectively orient one or more fields of view of one or more second cameras mounted on the UAV based on the indication of the scanned object, acquiring at least one image from the one or more positions using the one or more second cameras, and determining a 3D model based on distances to one or more surfaces of the scanned object determined based on the at least one image, where the 3D model includes a plurality of points corresponding to the surface of the scanned object.
[0126] Clause 7. A method including: determining, by one or more processors of an unmanned aerial vehicle (UAV), a plurality of contour paths spaced apart from one another along at least one axis associated with a scanned object, where each contour path is spaced apart from a surface of the scanned object based on a selected distance; determining a plurality of image acquisition locations for each contour path, where each image acquisition location indicates a respective location where a respective image of the surface of the scanned object is to be acquired; and navigating the UAV along the plurality of contour paths at the determined speed while acquiring images of the surface of the scanned object based on the image acquisition locations.
[0127] Clause 8. The method of clause 7, further including determining a maximum speed for traversing at least a portion of the plurality of contour paths based at least on the illumination associated with the surface and the selected distance, and navigating the UAV along the plurality of contour paths at the determined speed by traversing at least a portion of the plurality of contour paths at or below the maximum speed.
[0128] Clause 9. The method of clause 8, wherein determining the maximum velocity is further based on a threshold level of subject motion determined to be acceptable for the captured image.
[0129] Clause 10. The method of clause 7, further including determining a change in illumination associated with at least a portion of a surface to be scanned, and altering a velocity of the UAV based at least in part on determining the change in illumination.
[0130] Clause 11. The method of clause 7, further comprising determining an overlap of the images of the surface and determining a distance between adjacent image capture locations of each contour path based at least on the overlap, the selected distance, and a field of view of a camera used to capture the images of the surface.
[0131] Clause 12. The method of clause 7, further comprising determining a sidelap between the images of the surface and determining a distance between adjacent contour paths based at least on the sidelap, the selected distance, and a field of view of a camera used to acquire the images of the surface.
[0132] Clause 13. The method of clause 7, further comprising associating a coordinate system with the scan object based at least on a configuration of the scan object, the coordinate system including at least one axis, and the coordinate system is associated with the scan object based at least on at least one of aligning the at least one axis with at least one of a longest edge of the scan object or a longest element of the scan object.
[0133] Clause 14. The method of clause 7, further comprising navigating the UAV along at least a portion of the plurality of contour paths based on a constant velocity while acquiring images of the surface based on regular intervals.
[0134] Section 15. An unmanned aerial vehicle (UAV), comprising: a UAV body including a propulsion mechanism; a camera mounted to the UAV body; and one or more processors configured to perform operations according to executable instructions, the operations including: receiving an indication of an object to be scanned; determining a plurality of contour paths spaced apart from one another along at least one axis associated with the object to be scanned, where each contour path is spaced apart from a surface of the object based on a selected distance; determining a plurality of image acquisition locations for each contour path, where each image acquisition location indicates a location where an image of the surface of the object should be acquired; determining a maximum speed for traversing the plurality of image acquisition locations of the plurality of contour paths, where the maximum speed is based at least in part on the selected distance and on lighting associated with the surface; and operating the propulsion mechanism to navigate the UAV along the plurality of contour paths at a speed based at least in part on the maximum speed while acquiring images of the surface of the object using the camera based on the image acquisition location of each contour path of the contour paths.
[0135] Clause 16. The UAV of clause 15, wherein the operations further include determining an overlap of the images of the surface and determining a distance between adjacent image acquisition locations of each contour path based at least on the overlap, the selected distance, and a field of view of a camera used to acquire the images of the surface.
[0136] Clause 17. The UAV of clause 15, wherein the operations further include determining a side lap between the images of the surface and determining a distance between adjacent contour paths based at least on the side lap, the selected distance, and a field of view of a camera used to acquire the images of the surface.
[0137] Clause 18. The UAV of clause 15, wherein the operations further include determining a change in illumination associated with at least a portion of a surface to be scanned, and altering a velocity of the UAV based at least in part on determining the change in illumination.
[0138] Clause 19. The UAV of clause 15, wherein the operations further include, in response to receiving an indication of the scan target, determining positions of a plurality of points on a surface of the scan target relative to the UAV based on one or more images of the scan target, and associating a coordinate system with the scan target based at least on a configuration of the scan target determined from the plurality of points.
[0139] Clause 20. The UAV of clause 19, wherein the operations further include associating the selected axis of the coordinate system with at least one of a longest edge of the scanned object or a longest element of the scanned object.
[0140] Various instructions, processes, and techniques described herein may be considered in the general context of computer-executable instructions, such as computer programs and applications, stored on a computer-readable medium and executed by a processor herein. In general, the terms programs and applications may be used interchangeably and may include instructions, routines, modules, objects, components, data structures, executable code, and the like, for performing particular tasks or implementing particular data types. These programs, applications, and the like may be executed as native code or may be downloaded and executed in a virtual machine or other just-in-time compilation execution environment, and the like. Typically, the functionality of the programs and applications may be combined or distributed as desired in various implementations. Implementations of these programs, applications, and techniques may be stored on computer storage media or transmitted over some form of communication medium.
[0141] Although the present subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.
Claims
1. An unmanned aerial vehicle (UAV), a first camera mounted on the UAV; and one or more processors configured to perform operations according to the executable instructions, the operations including: determining a surface of a scan object to be scanned; determining a distance of the scanned object from the surface as a selected distance associated with acquiring an image; determining a maximum speed for traversing at least the first portion of the surface based at least on illumination associated with at least the first portion of the surface and based on the selected distance; and navigating the UAV at a speed based at least on the determined maximum speed relative to at least the first portion of the surface of the scanned object based on the selected distance while acquiring an image of at least the first portion of the surface of the scanned object.
2. The operation includes: acquiring the image of the first portion of the surface of the scanned object while flying at or below the maximum speed; determining that an illumination associated with a second portion of the surface of the scanned object is different from the illumination associated with the first portion of the surface; determining a maximum speed for traversing the second portion of the surface that differs from the maximum speed determined for traversing the first portion of the surface based at least on the illumination associated with the second portion of the surface and based on the selected distance; The UAV of claim 1 , further comprising:
3. The UAV of claim 1 , wherein the operation of determining the maximum speed is further based on a threshold level of subject blur determined to be acceptable for the acquired image.
4. The operation includes: determining a plurality of contour paths spaced apart from one another along at least one axis associated with the scan object, each contour path being spaced apart from the surface of the scan object based on the selected distance; determining a plurality of image acquisition locations for each contour path, each image acquisition location indicating a respective location at which an image of the surface of the scanned object should be acquired; Further comprising: The UAV of claim 1, wherein navigating the UAV relative to at least the first portion of the surface of the scanned object based on the selected distance while acquiring images of at least the first portion of the surface of the scanned object at a speed based on the determined maximum speed includes navigating the UAV along the one or more of the contour paths at a determined speed while acquiring images of the surface of the scanned object based on the image acquisition position.
5. The operation includes: associating a coordinate system with the scan object based at least on a configuration of the scan object, the coordinate system including the at least one axis; determining the plurality of spaced apart contour paths along the at least one axis based at least on respective contours of the surface of the scanned object; The UAV of claim 5, further comprising:
6. The operation includes: receiving an indication of the scan target from a computing device; causing the UAV to assume one or more positions for respectively directing one or more fields of view of one or more second cameras mounted on the UAV based on the indication of the scanned target; acquiring at least one image from the one or more locations using the one or more second cameras; determining a 3D model based on distances to one or more surfaces of the scanned object determined based on the at least one image, the 3D model including a plurality of points corresponding to the surfaces of the scanned object; The UAV of claim 1 , further comprising:
7. determining, by one or more processors of an unmanned aerial vehicle (UAV), a plurality of contour paths spaced apart from one another along at least one axis associated with a scan object, each contour path being spaced apart from a surface of the scan object based on a selected distance; determining a plurality of image acquisition locations for each contour path, each image acquisition location indicating a respective location on the scanned surface where a respective image is to be acquired; navigating the UAV along the plurality of contour paths at a determined speed while acquiring images of the surface of the scanned object based on the image acquisition locations; A method comprising:
8. determining a maximum speed for traversing at least a portion of the plurality of contour paths based at least on illumination associated with the surface and the selected distance; navigating the UAV along the plurality of contour paths at the determined speed by traversing at least the portion of the plurality of contour paths at or below the maximum speed; The method of claim 7 further comprising:
9. The method of claim 8 , wherein determining the maximum velocity is further based on a threshold level of subject motion determined to be acceptable for the captured image.
10. determining a change in illumination associated with at least a portion of the surface of the scanned object; Varying the velocity of the UAV based at least in part on determining the change in the illumination. The method of claim 7 further comprising:
11. determining an overlap of the images of the surface; determining a distance between adjacent image capture locations of each of the contour paths based at least on the overlap, the selected distance, and a field of view of a camera used to capture the images of the surface; The method of claim 7 further comprising:
12. determining a sidelap between the images of the surface; determining a distance between adjacent contour paths based at least on the side lap, the selected distance, and a field of view of a camera used to acquire the image of the surface; The method of claim 7 further comprising:
13. and associating a coordinate system with the scan object based at least on a configuration of the scan object, the coordinate system including the at least one axis, the coordinate system comprising: Aligning the at least one axis with at least one of a longest edge of the scanned object or a longest element of the scanned object. The method of claim 7 , wherein the scan target is associated with the scan target based at least on at least one of:
14. The method of claim 7 , further comprising navigating the UAV along at least a portion of the plurality of contour paths based on a constant speed while acquiring the images of the surface based on regular intervals.
15. An unmanned aerial vehicle (UAV), a UAV body including a propulsion mechanism; A camera attached to the UAV body; and one or more processors configured to perform operations according to the executable instructions, the operations including: receiving an indication of a scan target; determining a plurality of contour paths spaced apart from one another along at least one axis associated with the scan object, each contour path being spaced apart from a surface of the scan object based on a selected distance; determining a plurality of image acquisition locations for each contour path, each image acquisition location indicating a location at which an image of the scanned surface should be acquired; determining a maximum speed for traversing the plurality of image acquisition locations of the plurality of contour paths, the maximum speed being based at least in part on the selected distance and lighting associated with the surface; operating the propulsion mechanism to navigate the UAV along the plurality of contour paths at a speed based at least in part on the maximum speed while acquiring images of the surface of the scanned object using the camera based on the image acquisition location of each contour path among the contour paths; and unmanned aerial vehicles (UAVs).
Citation Information
Patent Citations
Information processing device, flight control instruction method, program, and recording medium
JP2020050261A
Image capturing device and image compositing device
WO2018212008A1
