Unmanned aerial vehicle-based semantic structure inspection

Abstract

Systems and techniques for unmanned aerial vehicle (UAV)-based semantic structure inspection are disclosed. In some implementations, a UAV performs operations to semantically understand components of a structure under inspection. In some implementations, semantic segmentation rendering is performed to encode structural data, including precise and relative component locations, in pixel values of an image depicting a structure. In some implementations, semantic three-dimensional scan is performed for the multi-phase inspection of a structure using a UAV.

Description

UNMANNED AERIAL VEHICLE-BASED SEMANTIC STRUCTURE INSPECTIONTECHNICAL FIELD

[0001] This application generally relates to structure inspection using an unmanned aerial vehicle (UAV), and, more specifically, to semantic three-dimensional (3D) scan systems and techniques for UAV-based semantic understanding for structure inspection.BACKGROUND

[0002] UAVs are often used to capture images from vantage points that would otherwise be difficult for humans to reach. Typically, a UAV is operated by a human using a controller to remotely control the movements and image capture functions of the UAV. In some cases, a UAV may have automated flight and autonomous control features. For example, automated flight features may rely upon various sensor input to guide the movements of the UAV.SUMMARY

[0003] Systems and techniques for, inter alia, UAV-based semantic understanding for structure inspection are disclosed.

[0004] In some implementations, a method comprises: obtaining images captured using a camera of an unmanned aerial vehicle during an exploration inspection of a structure; determining components of the structure based on the images and a taxonomy associated with the structure; generating a visual representation of the components; and outputting the visual representation of the components to a user device in communication with the unmanned aerial vehicle to enable selections of ones of the components for further inspection using the unmanned aerial vehicle.

[0005] In some implementations of the method, determining the components of the structure based on the images and the taxonomy associated with the structure comprises: performing a computer vision process to detect the components within the images; and identifying the components by object type using the taxonomy.

[0006] In some implementations of the method, identifying the components by the object type using the taxonomy comprises: using a machine learning model to process the detected components against the taxonomy.

[0007] In some implementations of the method, the method comprises: updating thetaxonomy based on the detected components using a machine learning model.

[0008] In some implementations of the method, the computer vision process includes at least one of object detection or image segmentation.

[0009] In some implementations of the method, the visual representation of the components includes a three-dimensional graphical representation of the structure and generating the visual representation of the components comprises: labeling respective portions of the three-dimensional graphical representation of the structure according to information associated with the components.

[0010] In some implementations of the method, labeling the respective portions of the three-dimensional graphical representation of the structure according to information associated with the components comprises: annotating a portion of the three-dimensional graphical representation corresponding to a component with an unfavorable status to indicate the unfavorable status.

[0011] In some implementations of the method, the visual representation of the components includes a hierarchical text representation of the structure and generating the visual representation of the components comprises: generating the hierarchical text representation of the structure according to an arrangement of the components within the taxonomy.

[0012] In some implementations of the method, the method comprises: obtaining, from the user device, user input indicating the selections of the ones of the components within a graphical user interface within which the visual representation of the components is output for display.

[0013] In some implementations, a UAV comprises: one or more cameras; one or more memories; and one or more processors configured to execute instructions stored in the one or more memories to: capture one or more images of a structure using the one or more cameras; determine components of the structure based on the images and a taxonomy associated with the structure; and output a visual representation of the components to a user device to enable selections of ones of the components for inspection.

[0014] In some implementations of the UAV, the components are detected based on a computer vision process performed against the one or more images and identified by object type using the taxonomy.

[0015] In some implementations of the UAV, the one or more processors are configured to execute the instructions to: generate the visual representation of the components based on the determination of the components.

[0016] In some implementations of the UAV, the visual representation includes one or both of a three-dimensional graphical representation of the structure or a hierarchical text representation of the structure.

[0017] In some implementations of the UAV, user input indicating the selections of the ones of the components is obtained from the user device to configure the unmanned aerial vehicle to perform an inspection of the selected ones of the components.

[0018] In some implementations, a system comprises: an unmanned aerial vehicle; and a user device in communication with the unmanned aerial vehicle, wherein the unmanned aerial vehicle is configured to: capture images of a structure during an exploration inspection of the structure; determine components of the structure based on the images and a taxonomy associated with the structure; and output, to the user device, a visual representation of the components to enable selections of ones of the components for further inspection.

[0019] In some implementations of the system, the components are determined based on a detection of the components by a computer vision process performed against the images and an identification of object types of the detected components within the taxonomy.

[0020] In some implementations of the system, the user device is configured to: render a graphical user interface that outputs the visual representation of the components for display; and obtain, via the graphical user interface, user input indicating the selections of the ones of the components.

[0021] In some implementations of the system, the unmanned aerial vehicle is configured to: perform a further inspection of the ones of the components based on the user input.

[0022] In some implementations of the system, the unmanned aerial vehicle is configured to: obtain a three-dimensional graphical representation of the structure; and generate the visual representation of the components by labeling respective portions of the three- dimensional graphical representation of the structure.

[0023] In some implementations of the system, the unmanned aerial vehicle is configured to: generate the visual representation of the components as a hierarchical text representation of the structure according to an arrangement of the components within the taxonomy.

[0024] Systems and techniques for, inter alia, semantic segmentation rendering for precise localization of structure components on captured images are disclosed.

[0025] In some implementations, a method comprises: obtaining images captured during an unmanned aerial vehicle-based exploration inspection of a structure; encoding, in each pixel value that corresponds to the structure within the images, identifiers of the structure, a component of the structure depicted using the pixel value, and a location of the component;segmenting pixel values of the images into polygons according to the encoded identifiers; and storing data indicative of the polygons for use in a further inspection of the structure.

[0026] In some implementations of the method, the method comprises: obtaining a three- dimensional graphical representation of the structure; and rendering, according to the encoded identifiers, the three-dimensional graphical representation of the structure using shaders that visually distinguish each component of the structure, wherein the data indicative of the polygons identifies respective ones of the shaders.

[0027] In some implementations of the method, obtaining the three-dimensional graphical representation of the one or more structures comprises: generating the three-dimensional graphical representation based on the images using a ray-based optimization technique.

[0028] In some implementations of the method, rendering the three-dimensional graphical representation of the structure using the shaders for each component of the structure according to the encoded identifiers comprises: generating a UV map of the structure according to the shaders; and rendering the three-dimensional graphical representation of the structure based on the UV map.

[0029] In some implementations of the method, each pixel value is represented by coordinates of the UV map.

[0030] In some implementations of the method, rendering the three-dimensional graphical representation of the structure using the shaders for each component of the structure according to the encoded identifiers comprises: rendering different surfaces of a component of the structure within the three-dimensional graphical representation using different ones of the shaders.

[0031] In some implementations of the method, segmenting the pixel values of the images into the polygons according to the encoded identifiers comprises: performing a panoptic segmentation process against the images to segment the pixel values into vector annotations corresponding to the polygons.

[0032] In some implementations of the method, the method comprises: for each component of the structure, determining a location of the component using a visual positioning system of the unmanned aerial vehicle and pose information of the unmanned aerial vehicle.

[0033] In some implementations of the method, the method comprises: obtaining a query for images depicting the structure of one or more structures; and using the stored data to indicate the images in response to the query.

[0034] In some implementations, a UAV comprises: one or more cameras; one or morememories; and one or more processors configured to execute instructions stored in the one or more memories to: capture images of a structure using the one or more cameras; render a three-dimensional graphical representation of the structure with shaders visually distinguishing components of the structure by encoding, in each pixel value that corresponds to the structure within the images, identifiers of the structure, a component of the structure depicted using the pixel value, and a location of the component; and storing data indicative of polygons associated with pixel values of the images for use in a further inspection of the structure.

[0035] In some implementations of the UAV, the one or more processors are configured to execute the instructions to: segment the pixel values of the images into the polygons according to the encoded identifiers.

[0036] In some implementations of the UAV, to segment the pixel values of the images into the polygons according to the encoded identifiers, the one or more processors are configured to execute the instructions to: perform a panoptic segmentation process against the images to segment the pixel values into vector annotations corresponding to the polygons.

[0037] In some implementations of the UAV, to render the three-dimensional graphical representation of the structure with the shaders, the one or more processors are configured to execute the instructions to: generate a UV map of the structure according to the shaders; and render the three-dimensional graphical representation of the structure based on the UV map.

[0038] In some implementations of the UAV, to render the three-dimensional graphical representation of the structure with the shaders, the one or more processors are configured to execute the instructions to: associate the pixel value with coordinates of the UV map of the structure.

[0039] In some implementations, a system comprises: an unmanned aerial vehicle; and a user device in communication with the unmanned aerial vehicle, wherein the unmanned aerial vehicle is configured to: render a three-dimensional graphical representation of a structure with shaders visually distinguishing components of the structure by encoding, in each pixel value that corresponds to the structure within images captured of the structure, identifiers of the structure, a component of the structure depicted using the pixel value, and a location of the component; and cause an output of data associated with the rendered three- dimensional graphical representation of the structure at the user device.

[0040] In some implementations of the system, the unmanned aerial vehicle is configured to: capture the images; and obtain the three-dimensional graphical representation of the structure.

[0041] In some implementations of the system, to obtain the three-dimensional graphical representation of the structure, the unmanned aerial vehicle is configured to: generate the three-dimensional graphical representation based on the images using graph clustering.

[0042] In some implementations of the system, the unmanned aerial vehicle is configured to: segment pixel values of the images into polygons according to the encoded identifiers; and store data indicative of the polygons for use in a further inspection of the structure.

[0043] In some implementations of the system, to segment the pixel values of the images into the polygons according to the encoded identifiers, the unmanned aerial vehicle is configured to: perform a panoptic segmentation process against the images to segment the pixel values into vector annotations corresponding to the polygons.

[0044] In some implementations of the system, the data indicative of the polygons identifies respective ones of the shaders.

[0045] Systems and techniques for, inter alia, multi-phase semantic 3D scan are disclosed.

[0046] In some implementations, a method comprises: performing, using an unmanned aerial vehicle, a first phase inspection of a structure to determine a semantic understanding of components associated with the structure and pose information of the components; determining, based on the semantic understanding of the components and the pose information, a flight path indicating capture points and camera poses associated with the capture points; and performing, using the unmanned aerial vehicle, a second phase inspection of a subset of the components according to the flight path.

[0047] In some implementations of the method, performing the first phase inspection of the structure to determine the semantic understanding of the components and the pose information comprises: navigating the unmanned aerial vehicle to a distance from the structure such that all of the structure is within a field of view of a camera of the unmanned aerial vehicle.

[0048] In some implementations of the method, performing the first phase inspection of the structure to determine the semantic understanding of the components and the pose information comprises: detecting the components using a camera of the unmanned aerial vehicle; and storing data indicative of the detected components on two-dimensional images associated with associated poses of the camera.

[0049] In some implementations of the method, detecting the components comprises: triangulating, using the unmanned aerial vehicle, locations of the components.

[0050] In some implementations of the method, determining the flight path indicating thecapture points and the camera poses comprises: determining a number of columns for the unmanned aerial vehicle to vertically navigate, wherein each column of the number of columns includes one or more of the capture points.

[0051] In some implementations of the method, the number of columns is four columns, the four columns form a rectangular boundary surrounding the structure, and performing the second phase inspection of the subset of the components according to the flight path comprises: navigating the unmanned aerial vehicle about the rectangular boundary including ascending to a traversal height while moving between ones of the four columns.

[0052] In some implementations of the method, the number of columns is two columns, the two columns are at diagonally opposing locations about the structure, and performing the second phase inspection of the subset of the components according to the flight path comprises: navigating the unmanned aerial vehicle between the two columns over the structure.

[0053] In some implementations of the method, performing the second phase inspection of the subset of the components according to the flight path comprises: while the unmanned aerial vehicle is at a capture point of the capture points, aiming a camera of the unmanned aerial vehicle at a component of the components according to a camera pose of the camera poses; and capturing, using the aimed camera, an image of the component.

[0054] In some implementations of the method, aiming the camera at the component according to the camera pose comprises: continuously attempting to detect the component within a video feed captured using the camera until the component is centered in images of the video feed.

[0055] In some implementations of the method, the method comprises: labeling the image with information associated with one or both of the structure or the flight path.

[0056] In some implementations of the method, the method comprises: obtaining user input corresponding to one or more of an object of interest, a traversal height, a flight distance, a maximum speed, an exploration radius, a gimbal angle, or a column path, wherein the user input is used to perform the first phase inspection.

[0057] In some implementations, a UAV comprises: one or more cameras; one or more memories; and one or more processors configured to execute instructions stored in the one or more memories to: perform a first phase inspection of a structure to determine, based on one or more images captured using the one or more cameras, a semantic understanding of components associated with the structure and pose information of the components; and perform a second phase inspection of a subset of the components according to a flight paththat is based on the components and the pose information.

[0058] In some implementations of the UAV, to perform the first phase inspection of the structure to determine the semantic understanding of the components and the pose information, the one or more processors are configured to execute the instructions to: capture the one or more images while all of the structure remains within a field of view of the one or more cameras; and perform image segmentation to detect the components within the one or more images.

[0059] In some implementations of the UAV, to detect the components within the one or more images, the one or more processors are configured to execute the instructions to: triangulate locations of the components to determine unique locations in three-dimensional space of the components.

[0060] In some implementations of the UAV, to perform the second phase inspection of the subset of the components according to the flight path, the one or more processors are configured to execute the instructions to: aim the one or more cameras at a component; and capture, using the aimed one or more cameras, an image of the component.

[0061] In some implementations of the UAV, the one or more processors are configured to execute the instructions to: determine the flight path based on the components and the pose information, wherein the flight path indicates capture points and camera poses associated with the capture points, and wherein the capture points are arranged into a number of columns for the unmanned aerial vehicle to vertically navigate.

[0062] In some implementations, a system comprises: an unmanned aerial vehicle; and a user device in communication with the unmanned aerial vehicle, wherein the unmanned aerial vehicle is configured to: perform a first phase inspection of a structure according to user input obtained from the user device to determine a semantic understanding of components associated with the structure and pose information of the components; determine, based on the semantic understanding of the components and the pose information, a flight path indicating a number of columns for the unmanned aerial vehicle to vertically navigate, wherein each column of the number of columns includes one or more capture points each associated with one or more of the components; and perform a second phase inspection according to the flight path.

[0063] In some implementations of the system, the first phase inspection is performed while all of the structure is within a field of view of a camera of the unmanned aerial vehicle and the components are determined based on images captured using the camera.

[0064] In some implementations of the system, the camera is aimed at a component of theone or more of the components according to a camera pose associated with a respective capture point of the one or more capture points to capture an image of the component during the second phase inspection.

[0065] In some implementations of the system, where the number of columns is four columns, the four columns form a rectangular boundary surrounding the structure and the unmanned aerial vehicle navigates about the rectangular boundary including ascending to a traversal height while moving between ones of the four columns, and where the number of columns is two columns, the two columns are at diagonally opposing locations about the structure and the unmanned aerial vehicle navigates between the two columns over the structure.BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The disclosure is best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, according to common practice, the various features of the drawings are not to-scale. On the contrary, the dimensions of the various features are arbitrarily expanded or reduced for clarity.

[0067] FIG. 1 is an illustration of an example of a UAV system.

[0068] FIG. 2A is an illustration of an example of a UAV as seen from above.

[0069] FIG. 2B is an illustration of an example of a UAV as seen from below.

[0070] FIG. 3 is an illustration of an example of a controller for a UAV.

[0071] FIG. 4 is an illustration of an example of a dock for facilitating autonomous landing of a UAV.

[0072] FIG. 5 is a block diagram of an example of a hardware configuration of a UAV.

[0073] FIG. 6 is a block diagram of an example of a UAV-based semantic understanding system.

[0074] FIG. 7 is a block diagram of example functionality of UAV-based semantic understanding software.

[0075] FIG. 8A is an illustration of an example of a structure inspected using a UAV.

[0076] FIG. 8B is an illustration of an example of a component listed generated according to a UAV-based semantic understanding of the structure.

[0077] FIG. 9 is a flowchart of an example of a technique for UAV-based semantic understanding for structure inspection.

[0078] FIG. 10 is a flowchart of an example of a technique for determining structure components without pre-existing semantic data.

[0079] FIG. 11 is a flowchart of an example of a technique for determining structure components with pre-existing semantic data.

[0080] FIG. 12 is a block diagram of an example of a semantic segmentation rendering system.

[0081] FIG. 13 is a block diagram of example functionality of semantic segmentation rendering software.

[0082] FIG. 14 is an illustration of an example of a structure and its components for which semantic output may be generated.

[0083] FIG. 15 is an illustration of an example of a UV map for a structure and encodings for a pixel value corresponding to a portion of the structure.

[0084] FIG. 16 is a flowchart of an example of a technique for semantic segmentation rendering for precise localization of structure components on captured images.

[0085] FIG. 17 is a block diagram of an example of a multi -phase semantic 3D scan system.

[0086] FIG. 18 is a block diagram of example functionality of multi-phase semantic 3D scan software.

[0087] FIG. 19 is a block diagram of an example workflow for multi-phase semantic 3D scan.

[0088] FIGS. 20A-B are illustrations of example columns of a flight path used during an inspection phase for semantic 3D scan.

[0089] FIG. 21 is a flowchart of an example of a technique for multi-phase semantic 3D scan for structure inspection.DETAILED DESCRIPTION

[0090] The versatility of UAVs has made their use in structural inspection increasingly common in recent years. Personnel of various industries operate UAVs to navigate about structures (e.g., buildings, towers, bridges, pipelines, and utility equipment) and capture visual media indicative of the statuses and conditions thereof. Initially, UAV inspection processes involved the manual-only operation of a UAV, such as via a user device wirelessly communicating with the UAV; however, automated approaches have been more recently used in which a UAV determines a target structure and performs a sophisticated navigation and media capture process to automatically fly around the structure and capture images and / or video thereof. In some such cases, these automated approaches may involve the UAV or a computing device in communication therewith performing a 3D scan of a target structure togenerate a high-fidelity 3D geometric reconstruction thereof as part of an inspection process. For example, modeling of the 3D geometric reconstruction may be provided to a UAV operator to enable the UAV operator to identify opportunities for a further inspection of the structure.

[0091] However, these 3D scan approaches, although representing meaningful improvements over more manual and semi-manual inspection processes, may not be suitable in all structure inspection situations. In particular, such approaches generally involve high- fidelity processing and thus use large amounts of input media generate the 3D geometric reconstruction. This ultimately involves the capture of large amounts of images or videos over a relatively long period of time. Moreover, the 3D geometric reconstruction is purely geometric in that the reconstruction resulting from the 3D scan approach is limited to geometries of the structure. In some situations, however, a UAV operator may not need high- fidelity data about all of a structure, but may instead want to focus on details related or otherwise limited to certain components of the structure. Similarly, the UAV operator may want to have a semantic understanding of the structure that goes beyond what geometries can convey. For example, the UAV operator may want to understand what the specific components of the structure are, distinguishing between similar components (e.g., first and second blades of a wind turbine). Finally, while a UAV may of course be manually operated to focus only on certain aspects of a structure, such manual operation is often labor intensive, solve, imprecise, and non-deterministic and thus offers limited value.

[0092] Implementations of this disclosure address problems such as these using UAV- based semantic understanding approaches for structure inspection. In particular, UAV-based semantic understanding according to the implementations of this disclosure includes a UAV performing an exploration inspection of a structure to capture one or more images of the structure and then using those one or more images along with a semantic artificial intelligence (Al) engine to semantically determine components of the structure. Representations of those components are presented within one or more graphical user interfaces (GUIs) rendered for display at a user device in communication with the UAV to enable a user of the user device to select ones of the components for a further inspection by the UAV. For example, one GUI may include a text-based component list while another may include a 3D graphical representation of the structure with component details indicated therein. The implementations of this disclosure thus address a user experience for UAV operators in which objects in an inspection scene can be semantically identified and labeled according to type and sub-type taxonomy information and thereafter presented to the user toaide in the inspection process for a given structure.

[0093] The scan approaches and the resulting 3D geometric reconstructions of structures as described above do not convey such semantic understandings. In particular, because they merely output geometric information for the structure, they do not identify individual components or distinguish between components of the same type. Thus, UAV operators today must perform some manual process to denote structure components from one another. Typically, this involves a time consuming, labor intensive, and error prone process of a UAV operator entering data in connection UAV-captured images depicting portions of a structure, such as to catalog which components are shown in which images. Previous attempts at automating this task have, however, suffered drawbacks due to the technical challenge of adding image-level metadata to captured images based on an intended capture plan given the importance of understanding location-specific component information. For example, one such attempt has used markers or identification plates visible within images to link the detection of those markers or identification plates with certain structures. However, this approach only enables the identification of a structure, and thus cannot result in a precise location of the structure or its components being determined, and it is also only applicable to a small number of structures that use these markers or identification plates.

[0094] Implementations of this disclosure address problems such as these using semantic segmentation rendering for precise localization of structure components on captured images. Semantic segmentation rendering includes encoding structural data in pixel values of an image, so that structure components and their relative locations can be identified by the image encodings themselves. A UAV performs an exploration inspection of a site which includes one or more structures to capture images of the one or more structures. For each image, each pixel value of the image has encoded therein identifiers associated with one or more of the structure it corresponds to, the component it corresponds to, and a location of that component. A 3D graphical representation of the site is updated using shaders to visually distinguish the detected components of the one or more structures according to the encoded identifiers, and semantic annotations corresponding to those shaders are stored for further inspection use by the same or a different UAV. Thus, according to the implementations of this disclosure, a UAV is able to process images captured in flight, automatically determine what structures and components are visible in the images as well as their precise locations, and integrate this visual data in a cloud database for use in further inspections of those structures.

[0095] Implementations of this disclosure address problems such as those described above using multi-phase semantic 3D scan approaches for structure inspection. In particular,multi-phase semantic 3D scan according to the implementations of this disclosure includes a two-phase inspection process in which, during a first phase inspection, a UAV determines a semantic understanding of components associated with a structure under inspection and pose information of the components and, during a second phase inspection, the UAV navigates according to a flight path determined based on the components and pose information to capture images of a subset of the components of the structure. The first phase inspection may be referred to generally as an exploration phase in which structure components of relevance are identified within the inspection scene. The second phase inspection may be referred to generally as an inspection phase in which a detailed inspection limited to those components of relevance is performed. The multi-phase semantic 3D scan approaches disclosed herein involve the unique identification, by type and location, of structure components using triangulation and camera pose information obtained via the first phase inspection. Thus, using the implementations of this disclosure, portions of a structure may be inspected in a time- and media-sensitive manner and in detail beyond what is conveyable by structure geometry.

[0096] Generally, the quality of a scan (i.e., an inspection or an operation of an inspection) being “semantic” refers to the scan incorporating information and processing to recognize detailed contextual information for a structure and components thereof. In particular, a UAV performing a semantic 3D scan for a structure inspection uses a machine learning model to generate a comprehensive semantic scene graph of components of a structure, either starting from a blank list of components or an empirically (e.g., templatized) determined default list of components. The scene graph information, the arrangement of which is guided using a taxonomy of object types and sub-types, enables the operator of the UAV to identify components of the structure for a further, detailed inspection using the UAV. The scene graph information thus readily identifies components of relevance to a given inspection from amongst a generally or near exhaustive listing of components scanned via the first phase inspection. This semantic understanding of structures and components enables valuable automations for UAV-based structure inspections, improving the accuracy of captured data and materially reducing time and media capture requirements of other scan approaches. For example, a semantic understanding of a structure component may indicate to the UAV information about the component (e.g., its type, location, material, etc.) and / or its relationship to the structure and / or other components thereof.

[0097] To describe some implementations in greater detail, reference is first made to examples of hardware and software structures used to implement a UAV-based semantic structure inspection system. FIG. 1 is an illustration of an example of a UAV system 100.The system 100 includes a UAV 102, a controller 104, a dock 106, and a server 108.

[0098] The UAV 102 is a vehicle which may be controlled autonomously by one or more onboard processing aspects or remotely controlled by an operator, for example, using the controller 104. The UAV 102 may be implemented as one of a number of types of unmanned vehicle configured for aerial operation. For example, the UAV 102 may be a vehicle commonly referred to as a drone but may otherwise be an aircraft configured for flight within a human operator present therein. In particular, the UAV 102 may be a multi-rotor vehicle. For example, the UAV 102 may be lifted and propelled by four fixed-pitch rotors in which positional adjustments in-flight may be achieved by varying the angular velocity of each of those rotors.

[0099] The controller 104 is a device configured to control at least some operations associated with the UAV 102. The controller 104 may communicate with the UAV 102 via a wireless communications link (e.g., via a Wi-Fi network, a Bluetooth link, a ZigBee link, or another network or link) to receive video or images and / or to issue commands (e.g., take off, land, follow, manual controls, and / or commands related to conducting an autonomous or semi-autonomous navigation of the UAV 102). The controller 104 may be or include a specialized device. Alternatively, the controller 104 may be or include a mobile device, for example, a smartphone, tablet, laptop, or other device capable of running software configured to communicate with and at least partially control the UAV 102.

[0100] The dock 106 is a structure which may be used for takeoff and / or landing operations of the UAV 102. In particular, the dock 106 may include one or more fiducials usable by the UAV 102 for autonomous takeoff and landing operations. For example, the fiducials may generally include markings which may be detected using one or more sensors of the UAV 102 to guide the UAV 102 from or to a specific position on or in the dock 106. In some implementations, the dock 106 may further include components for charging a battery of the UAV 102 while the UAV 102 is on or in the dock 106. The dock 106 may be a protective enclosure from which the UAV 102 is launched. A location of the dock 106 may correspond to the launch point of the UAV 102.

[0101] The server 108 is a remote computing device from which information usable for operation of the UAV 102 may be received and / or to which information obtained at the UAV 102 may be transmitted. For example, the server 108 may be used to train a learning model usable by one or more aspects of the UAV 102 to implement functionality of the UAV 102. In another example, signals including information usable for updating aspects of the UAV 102 may be received from the server 108. The server 108 may communicate with the UAV 102over a network, for example, the Internet, a local area network, a wide area network, or another public or private network.

[0102] In some implementations, the system 100 may include one or more additional components not shown in FIG. 1. In some implementations, one or more components shown in FIG. 1 may be omitted from the system 100, for example, the server 108.

[0103] An example illustration of a UAV 200, which may, for example, be the UAV 102 shown in FIG. 1, is shown in FIGS. 2A-B. FIG. 2A is an illustration of an example of the UAV 200 as seen from above. The UAV 200 includes a propulsion mechanism 202 including some number of propellers (e.g., four) and motors configured to spin the propellers. For example, the UAV 200 may be a quad-copter drone. The UAV 200 includes image sensors, including a high-resolution image sensor 204. This image sensor 204 may, for example, be mounted on a gimbal to support steady, low-blur image capture and object tracking. The UAV 200 also includes image sensors 206, 208, and 210 that are spaced out around the top of the UAV 200 and covered by respective fisheye lenses to provide a wide field of view and support stereoscopic computer vision. The image sensors 206, 208, and 210 generally have a resolution which is lower than a resolution of the image sensor 204. Additionally, the UAV 200 includes a number of arms 212 (e.g., four). The propulsion mechanisms 202 may be mounted to the arms 212. The arms 212 are attached the body 214 of the UAV 200. The propulsion mechanisms 202, the arms 212, and the body 214 may collectively be referred to as the frame of the UAV 200. The UAV 200 also includes other internal hardware, for example, a processing apparatus (not shown). In some implementations, the processing apparatus is configured to automatically fold the propellers when entering a dock (e.g., the dock 106 shown FIG. 1), which may allow the dock to have a smaller footprint than the area swept out by the propellers of the propulsion mechanism 202.

[0104] FIG. 2B is an illustration of an example of the UAV 200 as seen from below. From this perspective, three more image sensors 216, 218, and 220 arranged on the bottom of the UAV 200 may be seen. These image sensors 216, 218, and 220 may also be covered by respective fisheye lenses to provide a generally wide field of view and support stereoscopic computer vision. The various image sensors of the UAV 200 may enable visual inertial odometry (VIO) for high resolution localization and obstacle detection and avoidance. For example, the image sensors may be used to capture images including infrared data which may be processed for day or night mode navigation of the UAV 200. The UAV 200 also includes a battery in battery pack 224 attached on the bottom of the UAV 200, with conducting contacts 222 to enable battery charging. The bottom surface of the battery pack 224 may be a bottomsurface of the UAV 200.

[0105] FIG. 3 is an illustration of an example of a controller 300 for a UAV, which may, for example, be the UAV 102 shown in FIG. 1. The controller 300 may, for example, be the controller 104 shown in FIG. 1. The controller 300 may provide a user interface for controlling the UAV and reviewing data (e.g., images) received from the UAV. The controller 300 includes a touchscreen 302, a left joystick 304, and a right joystick 306. In the example as shown, the touchscreen 302 is part of a mobile device 308 (e.g., a smartphone) that connects to a controller attachment 310, which, in addition to providing additional control surfaces including the left joystick 304 and the right joystick 306, may provide range extending communication capabilities for longer distance communication with the UAV.

[0106] FIG. 4 is an illustration of an example of a dock 400 for facilitating autonomous landing of a UAV, for example, the UAV 102 shown in FIG. 1. The dock 400 may, for example, be the dock 106 shown in FIG. 1. The dock 400 includes a cradle 402 (i.e., landing surface) with a fiducial 404, charging contacts 406 for a battery charger, a box 408 in the shape of a rectangular box with a door 410, and a retractable arm 412.

[0107] The cradle 402 is configured to hold a UAV. The UAV may be configured for autonomous landing on the cradle 402. The cradle 402 has a funnel geometry shaped to fit a bottom surface of the UAV at a base of the funnel. The tapered sides of the funnel may help to mechanically guide the bottom surface of the UAV into a centered position over the base of the funnel during a landing. For example, corners at the base of the funnel may serve to prevent the aerial vehicle from rotating on the cradle 402 after the bottom surface of the aerial vehicle has settled into the base of the funnel shape of the cradle 402. For example, the fiducial 404 may include an asymmetric pattern that enables robust detection and determination of a pose (i.e., a position and an orientation) of the fiducial 404 relative to the UAV based on an image of the fiducial 404, for example, captured with an image sensor of the UAV.

[0108] The conducting contacts 406 are contacts of a battery charger on the cradle 402, positioned at the bottom of the funnel. The dock 400 includes a charger configured to charge a battery of the UAV while the UAV is on the cradle 402. For example, a battery pack of the UAV (e.g., the battery pack 224 shown in FIG. 2B) may be shaped to fit on the cradle 402 at the bottom of the funnel shape. As the UAV makes its final approach to the cradle 402, the bottom of the battery pack will contact the cradle 402 and be mechanically guided by the tapered sides of the funnel to a centered location at the bottom of the funnel. When the landing is complete, the conducting contacts of the battery pack may come into contact withthe conducting contacts 406 on the cradle 402, making electrical connections to enable charging of the battery of the UAV. The dock 400 may include a charger configured to charge the battery while the UAV is on the cradle 402.

[0109] The box 408 is configured to enclose the cradle 402 in a first arrangement and expose the cradle 402 in a second arrangement. The dock 400 may be configured to transition from the first arrangement to the second arrangement automatically by performing steps including opening the door 410 of the box 408 and extending the retractable arm 412 to move the cradle 402 from inside the box 408 to outside of the box 408.

[0110] The cradle 402 is positioned at an end of the retractable arm 412. When the retractable arm 412 is extended, the cradle 402 is positioned away from the box 408 of the dock 400, which may reduce or prevent propeller wash from the propellers of a UAV during a landing, thus simplifying the landing operation. The retractable arm 412 may include aerodynamic cowling for redirecting propeller wash to further mitigate the problems of propeller wash during landing. The retractable arm supports the cradle 402 and enables the cradle 402 to be positioned outside the box 408, to facilitate takeoff and landing of a UAV, or inside the box 408, for storage and / or servicing of a UAV.[OHl] In some implementations, the dock 400 includes a fiducial 414 on an outer surface of the box 408. The fiducial 404 and the fiducial 414 may be detected and used for visual localization of the UAV in relation the dock 400 to enable a precise landing on the cradle 402. For example, the fiducial 414 may encode data that, when processed, identifies the dock 400, and the fiducial 404 may encode data that, when processed, enables robust detection and determination of a pose (i.e., a position and an orientation) of the fiducial 414 relative to the UAV. The fiducial 414 may be referred to as a first fiducial and the fiducial 404 may be referred to as a second fiducial. The first fiducial may be larger than the second fiducial to facilitate visual localization from farther distances as a UAV approaches the dock 400. For example, the area of the first fiducial may be 25 times the area of the second fiducial.

[0112] The dock 400 is shown by example only and is non-limiting as to form and functionality. Thus, other implementations of the dock 400 are possible. For example, other implementations of the dock 400 may be similar or identical to the examples shown and described within U.S. Patent Application Serial No. 17 / 889, 991, filed August 31, 2022, the entire disclosure of which is herein incorporated by reference.

[0113] FIG. 5 is a block diagram of an example of a hardware configuration of a UAV 500, which may, for example, be the UAV 102 shown in FIG. 1. The UAV 500 includes a processing apparatus 502, a data storage device 504, a sensor interface 506, acommunications interface 508, propulsion control interface 510, a user interface 512, and an interconnect 514 through which the processing apparatus 502 may access the other components.

[0114] The processing apparatus 502 is operable to execute instructions that have been stored in the data storage device 504 or elsewhere. The processing apparatus 502 is a processor with random access memory (RAM) for temporarily storing instructions read from the data storage device 504 or elsewhere while the instructions are being executed. The processing apparatus 502 may include a single processor or multiple processors each having single or multiple processing cores. Alternatively, the processing apparatus 502 may include another type of device, or multiple devices, capable of manipulating or processing data. The processing apparatus 502 may be arranged into processing unit, such as a central processing unit (CPU) or a graphics processing unit (GPU).

[0115] The data storage device 504 is a non-volatile information storage device, for example, a solid-state drive, a read-only memory device (ROM), an optical disc, a magnetic disc, or another suitable type of storage device such as a non-transitory computer readable memory. The data storage device 504 may include another type of device, or multiple devices, capable of storing data for retrieval or processing by the processing apparatus 502. The processing apparatus 502 may access and manipulate data stored in the data storage device 504 via the interconnect 514, which may, for example, be a bus or a wired or wireless network (e.g., a vehicle area network).

[0116] The sensor interface 506 is configured to control and / or receive data from one or more sensors of the UAV 500. The data may refer, for example, to one or more of temperature measurements, pressure measurements, a global positioning system (GPS) data, acceleration measurements, angular rate measurements, magnetic flux measurements, a visible spectrum image, an infrared image, an image including infrared data and visible spectrum data, and / or other sensor output. For example, the one or more sensors from which the data is generated may include single or multiple of one or more of an image sensor 516, an accelerometer 518, a gyroscope 520, a geolocation sensor 522, a barometer 524, and / or another sensor. In some implementations, the accelerometer 518 and the gyroscope 520 may be combined as an inertial measurement unit (IMU). In some implementations, the sensor interface 506 may implement a serial port protocol (e.g., inter-integrated circuit (I2C) or serial peripheral interface (SPI)) for communications with one or more sensor devices over conductors. In some implementations, the sensor interface 506 may include a wireless interface for communicating with one or more sensor groups via low-power, short-rangecommunications techniques (e.g., using a vehicle area network protocol).

[0117] The communications interface 508 facilitates communication with one or more other devices, for example, a paired dock (e.g., the dock 106), a controller (e.g., the controller 104), or another device, for example, a user computing device (e.g., a smartphone, tablet, or other device). The communications interface 508 may include a wireless interface and / or a wired interface. For example, the wireless interface may facilitate communication via a Wi-Fi network, a Bluetooth link, a ZigBee link, or another network or link. In another example, the wired interface may facilitate communication via a serial port (e.g., RS-232 or universal serial bus (USB)). The communications interface 508 further facilitates communication via a network, which may, for example, be the Internet, a local area network, a wide area network, or another public or private network.

[0118] The propulsion control interface 510 is used by the processing apparatus to control a propulsion system of the UAV 500 (e.g., including one or more propellers driven by electric motors). For example, the propulsion control interface 510 may include circuitry for converting digital control signals from the processing apparatus 502 to analog control signals for actuators (e.g., electric motors driving respective propellers). In some implementations, the propulsion control interface 510 may implement a serial port protocol (e.g., I2C or SPI) for communications with the processing apparatus 502. In some implementations, the propulsion control interface 510 may include a wireless interface for communicating with one or more motors via low-power, short-range communications (e.g., a vehicle area network protocol).

[0119] The user interface 512 allows input and output of information from / to a user. In some implementations, the user interface 512 can include a display, which can be a liquid crystal display (LCD), a light emitting diode (LED) display (e.g., an organic light-emitting diode (OLED) display), or another suitable display. In some such implementations, the user interface 512 may be or include a touchscreen. In some implementations, the user interface 512 may include one or more buttons. In some implementations, the user interface 512 may include a positional input device, such as a touchpad, touchscreen, or the like, or another suitable human or machine interface device.

[0120] In some implementations, the UAV 500 may include one or more additional components not shown in FIG. 5. In some implementations, one or more components shown in FIG. 5 may be omitted from the UAV 500, for example, the user interface 512.

[0121] FIG. 6 is a block diagram of an example of a UAV-based semantic understanding system 600. The system 600 includes a UAV 602 and a structure 604 to be inspected usingthe UAV 602. The UAV 602 may, for example, be the UAV 102 shown in FIG. 1, the UAV 200 shown in FIGS. 2A-B, and / or the UAV 500 shown in FIG. 5. The structure 604 is a physical, 3D structural object that can be wholly or partially flown around by the UAV 602. Non-limiting examples of the structure 604 include buildings (e.g., residential, commercial, or industrial, including internal portions (e.g., warehouse aisles)), bridges (e.g., utility, walkway, or roadway), towers (e.g., powerline transmission, radio, cellular, or water), energy equipment (e.g., sub-stations, wind turbines, solar cells or farms, or power plant cooling towers), vehicles (e.g., aircraft, marine vessels, spacecraft, or land vehicles), civil spaces (e.g., parks or parking lots), and geographic features (e.g., forests, bluffs, mountains, valleys, or ravines). In some cases, the structure 604 may include a living being rather than or in addition to an object as described above. Non-limiting examples of the living being in such a case include humans, animals, fungi, and plants.

[0122] The UAV 602 includes hardware and software that configure the UAV 602 to determine a semantic understanding of the structure 604 via an exploration inspection thereof. In particular, and in addition to other components as are described with respect to FIG. 5, the UAV 602 includes UAV-based semantic understanding software 606 and one or more cameras 608. The UAV-based semantic understanding software 606, which will be described further in connection with FIG. 7, includes functionality for enabling the UAV 602 to perform an exploration inspection of the structure 604 to capture one or more images of the structure 604 using the one or more cameras 608, determine components 610 of the structure 604, and communicate output indicative of the components 610 to a user device 612 via one or more semantic understanding GUIs that are output for display at the user device 612.

[0123] The components 610 generally are, include, or otherwise refer to components (e.g., objects, elements, pieces, equipment, sub-equipment, tools, or other physical matter) on or within the structure 604. In one non-limiting example, where the structure 604 is a powerline transmission tower, the components 610 may include one or more of an insulator, a static line or connection point, a conductor or overhead wire, a footer, or a transformer). The structure 604 may include any number of types of the components 610 and any number of ones of the components 610 for each of the individual types thereof.

[0124] The output of the exploration inspection of the structure 604 includes at least one or more images, captured using the one or more cameras 608, of the components 610. The output may be communicated to the user device 612 via a user / UAV interface. The user device 612 is a computing device configured to communicate with the UAV 602 wirelessly or by wire. For example, the user device 612 may be one or more of the controller 104 shown inFIG. 1 or the controller 300 shown in FIG. 3. In some cases, the UAV performs the exploration inspection of the structure 604 using input obtained from the user device 612. For example, the input obtained from the user device 612 may include information specifying one or more of the components 610 to inspect, a height at which the UAV 602 will fly while traversing along a flight path, a distance for the UAV 602 to fly, a maximum speed for the UAV 602 to fly, a radius that the UAV 602 will use to orbit or otherwise explore and evaluate the structure 604, a gimbal angle for the one or more cameras 608, a column path for the flight path, an instruction for data capture at waypoints of the flight path, or an instruction for stopping the inspection of the structure 604.

[0125] In some cases, the user of the user device 612 may specify one or more configurations for controlling the capture of some or all of the one or more images during the exploration inspection. For example, user input may be obtained from the user device 612 (e.g., via the semantic understanding GUIs 614) to configure the UAV to capture a single image for each component of the structure, to capture N images of the structure each from a different manually-specified or automatically-determined angle (i.e., where N is an integer greater than 1), to capture images from a specified ground sampling distance (GSD) (e.g., using the same GSD for the entire exploration inspection or using different GSDs for different portions of the structure), or the like.

[0126] The UAV 602, via the UAV-based semantic understanding software 606, may utilize empirical and / or machine learning-based data modeled for use in structure inspections. In particular, the UAV 602 may communicate with a server 616 that includes a data library 618 usable by the UAV-based semantic understanding software 606 to determine a semantic understanding of the structure 604. The server 616 is a computing device remotely accessible by or otherwise to the UAV 602. The data library 618 may, for example, include one or more of historic inspection data for the structure 604 or like structures, machine learning models (e.g., classification engines comprising trained convolutional or deep neural networks) trained according to inspection image output data sets with user-specific information culled, and / or other information usable by the system 600. In some cases, the data library 618 or other aspects at the server 616 may be accessed by or otherwise using the user device 612 instead of the UAV 602.

[0127] To further describe functionality of the UAV-based semantic understanding software 606, reference is next made to FIG. 7, which is a block diagram of example functionality of the UAV-based semantic understanding software 606. The UAV-based semantic understanding software 606, as described above, may use various functionality of aUAV (e.g., the UAV 602 shown in FIG. 6) to cause the UAV to obtain (e.g., generate, determine, etc.) a semantic understanding of a structure under inspection. The UAV-based semantic understanding software 606 includes a structure component determination tool 700, a component list generation tool 702, a GUI generation tool 704, and a user input processing tool 706. The tools 700 through 706 represent various functionality of the UAV-based semantic understanding software 606 and are non-limiting as to a particular structure or other expression of code, script, or the like.

[0128] The structure component determination tool 700 determines components on or within the structure to inspect and pose information of the components. During an exploration inspection, the UAV, using one or more cameras thereof, captures one or more images (i.e., exploration images) of the structure while navigating about an exploration path around some or all of the structure. The structure component determination tool 700 obtains, as input, those one or more images captured during the exploration inspection and processes them to determine, as output, components of the structure and the pose information of the components. In particular, the structure component determination tool 700 processes the one or more images captured using the UAV to determine relevant semantics for the components depicted in the one or more images. For example, the structure component determination tool 700 may use an Al engine, such as a classification engine (e.g., a convolutional or deep neural network), to determine semantics such as component type, facet identifier (e.g., based on a triangulation of the component), coordinates, material type, and the like. The output of such processing is metadata with which the respective images may be tagged and which may be indexed for future use with the UAV operator or an entity associated therewith.

[0129] The structure component determination tool 700 processes an image captured by the UAV during the exploration inspection to detect one or more objects, as one or more components, within the image and to determine pose information of those one or more components using navigation system information of the UAV. Detecting the one or more objects within the image as the one or more components includes performing object detection against the image to identify a bounding box for each component within the image. Objects detected within the image are thus represented via their bounding box, and object recognition is performed against the bounded objects to identify them as components of the structure, as well as to identify what components they are. In some cases, performing object recognition includes comparing objects detected within the images against modeled object data to determine whether an object appears as expected. The objects detected and recognized are identified as components and information indicating the components is stored in connectionwith the bounding boxes of the components. In some cases, image segmentation may be performed instead of object detection and object recognition. In some cases, other computer vision techniques may be performed instead of either of the above approaches.

[0130] The structure component determination tool 700 in particular leverages a taxonomy of structure components by object type and sub-type. For example, the taxonomy may include a hierarchical organization of structures and their components in which known structures for a given component are nested within a level underneath the structure entity and types and variations of those given components are nested underneath the level that shows the components. In some cases, a current state of the taxonomy may be maintained on the UAV or accessed by the structure component determination tool 700 (e.g., from a server, such as the server 616 shown in FIG. 6). In some cases, the taxonomy may instead be generated for use with the structure component determination tool 700 using a machine learning model trained based on structure inspection data. For example, such taxonomies may accordingly in such cases be generated on case-by-case bases using the images captured during respective exploration phases of structures to represent the specific object types and sub-types of those structures, rather than also including object types and sub-types that are irrelevant to those structures. The taxonomy in any case is used to identify names or identifiers of component which may be detected using the structure component determination tool 700.

[0131] As will be described below with respect to FIGS. 10 and 11, the components may be determined with or without pre-existing semantic data. In some cases, the machine learning model may translate components of the structure to a JavaScript Object Notation (JSON) format readable and usable by other aspects of the UAV-based semantic understanding software 606. In some cases, the components determined for the structure may be represented by the machine learning model as learnt features (e.g., embeddings) rather than in connection with a discrete taxonomy.

[0132] Determining the pose information of the one or more components detected within the image using the navigation system information of the UAV includes determining orientations and / or locations of the detected components based on their bounding boxes. The pose information of the components thus corresponds to or otherwise represents the orientations and / or locations of the different components within a scene of the structure. In particular, the pose information of a given component may identify sides, facets, surfaces, or like qualities of the component independently and with relation to others. For example, the pose information may identify a front of a component such that a location of the front of the component may be identifiable. In some cases, the pose information of a component may bebased on the type of the component. For example, because inspectors of powerline transmission towers generally need to view insulators from their top and bottom sides, the pose information for an insulator component may include a top pose above the insulator and a bottom pose below the insulator.

[0133] Because it is important that objects be uniquely identified to prevent multiple of the same component from being confused as being the same exact component, the structure component determination tool 700, as part of the component and pose information determination, performs a triangulation process to ensure that specific component location information is known. In particular, locations of bounding boxes for detected components may be denoted using location data according to one or more of a visual positioning system (VPS) of the UAV, GPS, or another location-based system. The location data for bounding boxes of the detected components may be compared to determine duplicate components, in which a duplicate component is a component that has already been detected in a previous image. Duplicate components are accordingly culled to prevent them from being considered for scanning multiple times.

[0134] In some cases, the structure component determination tool 700 may detect that some object is present within an image captured using the UAV, but may be unable to determine what type of component that object is. In such a case, the structure component determination tool 700 may use a volume rendering technique such as Gaussian splatting to identify the object as a component based on its triangulated location. This volume rendering technique enables the structure component determination tool 700 to determine a location and shape in 3D space of the object. From there, the structure component determination tool 700 may use an Al engine to classify the object according to its location and shape. For example, the Al engine may be a classification engine (e.g., a convolutional or deep neural network) trained to predict object types according to their shapes and relative positionings on structures and nearby other object types. The Al engine may, for example, be trained according to structure and component data determined for historic inspections using UAVs, with userspecific information culled prior to the training.

[0135] In some implementations, the structure component determination tool 700 can determine a status of a given component. For example, the status may indicate an operational condition of the component or a damaged or other unfavorable condition of the component. The structure component determination tool 700 may determine the status for a component by using an Al engine, such as a machine learning model trained for image analysis and modeling, to compare a depiction of the component within an image captured using the UAVagainst operational and unfavorable (e.g., damaged) conditioned versions of like components. Where a component is determined to have an unfavorable condition, the status of the component may be flagged to the user of the user device, for example, for the user to determine whether to select the component for further inspection using the UAV, as described below.

[0136] The component list generation tool 702 generates, as a semantic scene graph for the structure, a component list indicating the components determined using the structure component determination tool 700. The components list is generally expressed as a scenegraph hierarchy in which components are represented (e.g., in text) in levels according to their relationship with other components and the structure itself. For example, the hierarchy of the components list may have a number of first level components that each correspond to a different area of the structure, each of the first level components may have a number of second level components that are considered children of the respective first level component, and so on. For example, where the structure is a house, the first level components may be components such as roof, sides (i.e., exterior walls), yards, and the like; the second level components of the roof may be components such as shingles, solar panels, chimneys, and the like; and so on.

[0137] In some implementations, the components list may instead be expressed as an object-type hierarchy in which the groupings of levels of the hierarchy, instead of corresponding to areas of the structure, correspond to types of objects. For example, referring again to an example in which the structure is a house, a first level entry of the object-type hierarchy may be windows, second level entries that are children of that first level entry may be different types of windows (e.g., arched, casement, sliding, etc.), and third level entries that are children of individual ones of those second level entries may be the actual components of the structure that were identified within one or more images captured using the UAV. In some implementations, the components list may be separately expressed within both a scene-graph hierarchy and an object-type hierarchy and the user may selectively navigate between them at the user device.

[0138] The GUI generation tool 704 generates instructions for rendering one or more GUIs at the user device in communication with the UAV. The GUIs are configured to display information associated with the components listed according to a component list generated using the component list generation tool 702. In particular, the GUI generation tool 704 may generate instructions configured to cause the user device to render a GUI that outputs the component list for display. For example, the GUI to render according to the instructions mayoutput the component list for display in a text format in which entries of the component list (i.e., all components, as in the case of a scene-graph hierarchy, or object type entries and components, as in the case of an object-type hierarchy) are represented in text. In some cases, status information for ones of the components may be included in the component list to indicate, for example, components which may be damaged and thus require further inspection.

[0139] The GUI may further include interactive user interface elements. For example, the interactive user interface elements may be checkboxes located proximate to the representations of the components to enable a user of the user device at which the GUI is output for display to interact with ones of the interactive user interface elements to select ones of the components within the GUI. In another example, the interactive user interface elements may include elements for navigating or manipulating the GUI, such as for expanding or collapsing groups of components (e.g., by first level component or entry), switching between GUIs, or modifying content of the GUI (e.g., of the component list), as described below. In some cases, multiple types of these example interactive user interface elements may be concurrently used.

[0140] The GUI generation tool 704 may further generate instructions configured to cause the user device to render one or more other GUIs, as well. For example, the GUI generation tool 704 may generate instructions for rendering a GUI which will output for display a graphical representation of the structure in a two-dimensional (2D) or 3D format. The graphical representation of the structure depicts the structure as shown in the one or more images captured using the UAV during the exploration inspection. For example, the GUI generation tool 704 may generate a 3D graphical representation of the structure using a 3D map of the structure obtained as output of a 3D scan or manual flight of the structure and / or using known 3D modeling techniques (e.g., by combining 2D images, generating a point cloud representation, importing known modeling data such as via computer-aided design (CAD) systems, etc.). In another example, the GUI generation tool 704 may generate a 2D graphical representation of the structure either by first generating a 3D graphical representation and then flattening same to a 2D surface or by reconstructing data captured within one of the exploration inspection images. Such a 3D or 2D graphical representation may, along with or in place of the component list described above, be considered a semantic scene graph for the structure under inspection. In some cases, the status of the components may be annotated within the graphical representation of the structure, for example, using coloring, text, highlighting, or another visual manner. In some implementations, the GUIgeneration tool 704 may generate instructions for the GUI of the graphical representation of the structure instead of a GUI of a component list.

[0141] The GUI which includes the graphical representation of the structure may include semantic viewing features for presenting labeled semantic information against the graphical representation. For example, the GUI may enable, via the rendering instructions generated using the GUI generation tool 704, the learning and adaptation of component information to introduce additional structured data within the graphical representation. Non-limiting examples of such structured data may include data obtained from an external modeling system (e.g., a CAD file) associated with the structure or construction plans obtained via an external building system. The structured data may be used by the Al engine to improve semantic understandings of some or all components of the structure, for example, by updating labeling data determined as inconsistent with the structured data.

[0142] The user input processing tool 700 obtains and processes input obtained from an operator of the UAV for the semantic understanding of the structure (e.g., the structure 604 shown in FIG. 6). The input includes configuration information for the scan and may be obtained in one or more formats, including free-text, selections from listed options, or the like. For example, the input may include information specifying one or more of objects of interest for the scan (e.g., the components to inspect, such as the components 610 shown in FIG. 6), a height (i.e., altitude) at which the UAV will fly while traversing along a flight path (e.g., between columns of the flight path, as described below), a distance for the UAV to fly from a boundary of the structure, a maximum speed for the UAV to fly (e.g., in miles per hour or kilometers per hour), a radius that the UAV will use to orbit or otherwise explore and evaluate the structure during a first phase inspection of the structure (e.g., in which the radius is larger than the sum of a stand-off distance and half of a dimensional measurement associated with the structure), a gimbal angle for one or more cameras of the UAV (e.g., set for individual objects or universally for all objects, in which the gimbal angle may by default be zero and in which multiple gimbal angles may accordingly be set), a column path for the flight path (e.g., indicating whether to use two or four columns), an instruction for data capture at waypoints of the flight path (e.g., enabling or disabling the automatic image capture at each waypoint of the flight path during autonomous flight), or an instruction for stopping the inspection of the structure (e.g., enabling or disabling the UAV pausing at each waypoint of the flight path during autonomous flight). The user input processing tool 700 processes this input and passes same for use with other tools of the UAV-based semantic understanding software 606 to configure the scan of the structure.

[0143] Additional user input may be obtained and processed using the user input processing tool 700 following an exploration inspection of the structure. In particular, one or more GUIs may be rendered at the user device according to instructions generated using the GUI generation tool 704. In one non-limiting example, such a GUI may include a component list generated using the component list generation tool 702. The component list may include individual components or entries each corresponding to a different component type. The components or other entries of the component list may each have a selectable or otherwise interactive element (e.g., a checkbox). When such an interactive element is selected or otherwise interacted, the additional user input is obtained, in which the additional user input indicates an intention of the UAV operator to cause the respective components associated with the subject entry to be inspected during a further inspection of the structure.

[0144] In some cases, a user of the user device may modify one or more aspects of a component identified as part of the structure. For example, the structure component determination tool 700 may at some point misidentify a component according to a visual similarity with a different component type or an obfuscation of the component by some other content within an image of the structure. In such a case, the user of the user device may correct the representation of the component via user input obtained and processed using the user input processing tool 706. In another example, the structure component determination tool 700 may accurately identify a component, but the user of the user device may prefer that the component be represented using a different identifier (e.g., due to a policy of a company or other entity with which the user of the user device is associated). In such a case, the user of the user device may change the representation of the component via user input obtained and processed using the user input processing tool 706.

[0145] The UAV-based semantic understanding software 606 is shown and described as being run (e.g., executed, interpreted, or the like) at a UAV used to perform an inspection of a structure. However, in some cases, the UAV-based semantic understanding software 606 or one or more of the tools 700 through 706 may be run other than at the UAV. For example, one or more of the tools 700 through 706 may be run at a user device or a server device and the output thereof may be communicated to the UAV for processing. In some cases, the UAV- based semantic understanding software 606 may include other tools beyond the tools 700 through 706. For example, the UAV-based semantic understanding software 606 may include a taxonomy update tool that uses a machine learning model to update a taxonomy used by the structure component determination tool 702 to determine the components of the structure.

[0146] To further describe some implementations in greater detail, reference is next madeto examples of illustrations of UAV-based semantic understanding content, which will be referenced in regard to an example use case of the UAV-based semantic understanding system 600 shown in FIG. 6. FIG. 8A is an illustration of an example of a structure 800 inspected using a UAV. FIG. 8B is an illustration of an example of a component list 824 generated according to a UAV-based semantic understanding of the structure 800 shown in FIG. 8A.

[0147] Referring first to FIG. 8A, the structure 800 is shown by example as a house (i.e., a detached, single-family residence). A UAV (e.g., the UAV 602) takes off in front of the structure 800 and begins capturing a live video feed using one or more cameras thereof. The live video feed is rendered within a GUI output for display a user device in communication with the UAV (e.g., the user device 612). A user of the user device (i.e., an operator of the UAV) initiates an exploration inspection of the structure 800 by the UAV by interacting with a user interface element within the GUI at the user device. The UAV then performs an exploration inspection of the structure 800 by orbiting around the structure 800 one or more times while capturing one or more images of the structure 800. In particular, the UAV captures one or more images depicting each side of the structure 800 to ensure all components of the structure 800 are pictured somewhere. Accordingly, the one or more images will be processed (e.g., using object detection and segmentation) to identify components of the structure 800 including a first roof facet 802 which includes shingles 804 and a solar panel 806, a second roof facet 808 which includes shingles 810 and a chimney 812, a first side wall 814 which includes a window 816 and a door 818, and a second side wall 820 which includes a window 822.

[0148] Referring next to FIG. 8B, the components list 824 generated for the structure 800 based on the exploration inspection described above is shown. The components list 824 is a hierarchical representation of the components of the structure 800, determined based on the processing of the one or more images captured during the exploration inspection, and thus shows representations (e.g., text names or like identifiers) for each of those components. As shown, the components list 824 includes representations of each of the components 802 through 822, as shown in FIG. 8A. The components list 824 may further include status information for some or all of the components 802 through 822. In the example shown, damage was detected with respect to the shingles 804 based on the processing of the one or more images depicting the shingles 804. The components list 824 accordingly includes a status identifier 826 indicating a damaged status of the shingles 804. The components list 824 includes interactive user interface elements next to the listed components 802 through 822. In the example shown, the interactive user interface elements are checkboxes. The user of theuser device interacts with ones of the interactive user interface elements to select corresponding ones of the components 802 through 822 for the UAV to focus on during a subsequent inspection scan of the structure 800. In some cases, given the hierarchical nature of the components list 824, the selection of a component (e.g., via checking the checkbox associated therewith) will cause an automated selection of any components nested thereunder (i.e., in a next child level of the hierarchy, in which the selected component is considered on a parent level of that child level). In the example shown, the user of the user device has selected the roof facet 1 802, the roof facet 2 808, the window 816, and the window 822, as well as the front yard (for which components are not shown). After selecting the applicable components, the user interacts with a user interface element 828 (e.g., a button) to cause an inspection of the selected components.

[0149] In some cases, a graphical representation of the structure 800 showing locations of ones of the components 802 through 822 may be accessed, viewed, and interacted with via the user device. For example, the user of the user device may interact with a representation of a component within the components list 824 to select to view the component on the graphical representation of the structure 800. In doing so, the user device outputs for display the graphical representation with a visual indicator for the component at an applicable location on the structure 800, within the same or a separate GUI as the GUI which outputs the components list 824. The graphical representation of the structure 800 may depict the structure 800 in a 2D or 3D view. In some cases, the user of the user device may interact with a respective component within the graphical representation of the structure 800 to select it for inspection by the UAV as if it had been selected (e.g., had its box checked) in the components list 824. The structure 800 as depicted in FIG. 8A may, for example, correspond to a 3D model or 3D graphical representation of the house described therewith.

[0150] To further describe some implementations in greater detail, reference is next made to examples of techniques for UAV-based semantic understanding. FIG. 9 is a flowchart of an example of a technique 900 for UAV-based semantic understanding for structure inspection. FIG. 10 is a flowchart of an example of a technique 1000 for determining structure components without pre-existing semantic data. FIG. 11 is a flowchart of an example of a technique 1100 for determining structure components with pre-existing semantic data.

[0151] The technique 900, the technique 1000, and / or the technique 1100 can be executed using computing devices, such as the systems, hardware, and software described with respect to FIGS. 1-8B. The technique 900, the technique 1000, and / or the technique 1100 can be performed, for example, by executing a machine-readable program or other computer-executable instructions, such as routines, instructions, programs, or other code. The steps, or operations, of the technique 900, the technique 1000, and / or the technique 1100, or another technique, method, process, or algorithm described in connection with the implementations disclosed herein can be implemented directly in hardware, firmware, software executed by hardware, circuitry, or a combination thereof.

[0152] For simplicity of explanation, the technique 900, the technique 1000, and the technique 1100 are each depicted and described herein as a series of steps or operations. However, the steps or operations in accordance with this disclosure can occur in various orders and / or concurrently. Additionally, other steps or operations not presented and described herein may be used. Furthermore, not all illustrated steps or operations may be required to implement a technique in accordance with the disclosed subject matter.

[0153] Referring first to FIG. 9, the technique 900 for UAV-based semantic understanding for structure inspection is shown. At 902, images of a structure under inspection are obtained. The images are captured using one or more cameras of a UAV during an exploration inspection of the structure. For example, the exploration inspection can include the UAV orbiting around the scene of the structure while capturing the images. Orbiting around the scene may, for example, include the UAV circling the structure, navigating along a perimeter or like boundary of the structure, or otherwise exploring the scene of the structure from a distance. In some cases, the orbiting can follow a zig-zag, lawnmower, or other pattern around the scene of the structure. The orbiting may be performed at one or more altitudes relative to the structure.

[0154] At 904, components of the structure are determined based on the images and a taxonomy. Determining the components of the structure includes performing a computer vision process to detect the components within the images and identifying the components by object type using the taxonomy. The computer vision process may, for example, be one or both of object detection or image segmentation. The taxonomy is associated with the structure. For example, the taxonomy may be generically created prior to the exploration inspection of the structure as having relevance to multiple structure types or it may be specifically generated for the inspection of the structure. In either such case, determining the components may include updating the taxonomy based on the detected components (e.g., using a machine learning model, such as a deep learning-based classification model trained based on historical structural inspection data). Where the taxonomy includes pre-existing information associated with a structure type of the structure, determining the components, and, in particular, identifying the components using the taxonomy may include accessing thetaxonomy and using a machine learning model trained based on historical structural inspection data, to process the detected components against the taxonomy.

[0155] At 906, a visual representation of the components (i.e., a semantic scene graph of the structure) is generated. The visual representation may, for example, be a 3D graphical representation of the structure or a hierarchical text representation of the structure. For example, where the visual representation is a 3D graphical representation of the structure, generating the visual representation may include labeling respective portions of the 3D graphical representation of the structure according to information associated with the components. In another example, where the visual representation is a hierarchical text representation of the structure, generating the visual representation may include generating the hierarchical text representation of the structure according to an arrangement of the components within the taxonomy (e.g., using nested levels of components or other entries). In some cases, multiple visual representations (e.g., a three-dimensional graphical representation of the structure and a hierarchical text representation of the structure) may be generated. In some cases, a status of a component may be indicated within the visual representation. For example, a portion of the 3D graphical representation or a portion of the hierarchical text representation that corresponds to a component with an unfavorable status may be annotated to indicate the unfavorable status.

[0156] At 908, the visual representation of the components is output to a user device in communication with the UAV. The visual representation is output to the user device to enable selections of ones of the components for further inspection using the UAV. For example, outputting the visual representation to the user device can include transmitting, to the user device, instructions for rendering one or more GUIs at the user device in which the GUIs are configured, based on the instructions, to output the visual representation for display.

[0157] At 910, user input indicating selections of ones of the components for further inspection is obtained via the user device. The user input may be obtained via one or more interactive user interface elements or like components of the one or more GUIs output for display at the user device. The user input is obtained to configure the UAV to perform an inspection of the selected ones of the components, in which the inspection focuses on those selected ones of the components. The inspection may, for example, be performed using the UAV according to a flight path determined based on locations of the selected ones of the components. The output of such inspection may, for example, include one or more images of those selected ones of the components.

[0158] Referring next to FIG. 10, the technique 1000 for determining structurecomponents without pre-existing semantic data is shown. The technique 1000 may, for example, be performed as or as part of the determination of the components as described above in connection with the technique 900 shown in FIG. 9. At 1002, a 3D map of a structure under inspection is obtained. Obtaining the 3D map of the structure can include generating the 3D map using images captured using a camera of a UAV during an exploration inspection of the structure by the UAV. Alternatively, obtaining the 3D map of the structure can include importing the 3D map (e.g., via a CAD or like system) from an external source or generating the 3D map using data obtained from an external source.

[0159] At 1004, a first-pass scene graph of the structure is generated based on the 3D map. The first-pass scene graph is a graphical representation of the structure. For example, the first-pass scene graph may be a graphical representation generated based on the components detected as being of the structure. In particular, generating the first-pass scene graph based on the 3D map includes performing a computer vision process against images captured using the UAV to detect the components of the structure. The computer vision process may, for example, be one or both of object detection or image segmentation.

[0160] At 1006, component labels are generated for the scene graph. The component labels are or otherwise include or represent data associated with structure components detected via the computer vision process. For example, a component label may include a name or other identifier for a component or other information usable to identify the component. Generating the component labels can include obtaining user input (e.g., from a user device in communication with the UAV) indicating the content of a label to generate for a content or using a machine learning model (e.g., trained based on historical structure inspection data) to generate such content for a label.

[0161] At 1008, component detection is performed against the scene graph according to the component labels. Performing the component detection against the scene graph according to the component labels includes using an Al engine (e.g., a classification-based machine learning model) to determine components to which the component labels apply. For example, where the user input indicating content of a label to generate is received to update the name of a component on a bounding box of that component, the Al engine can automatically detect other instances of that object type and accordingly update labels for those corresponding components. The labeled components of the scene graph may then be output as determined components.

[0162] Referring next to FIG. 11, the technique 1100 for determining structure components with pre-existing semantic data is shown. The technique 1100 may, for example,be performed as or as part of the determination of the components as described above in connection with the technique 900 shown in FIG. 9. At 1102, a 3D map of a structure under inspection is obtained. Obtaining the 3D map of the structure can include generating the 3D map using images captured using a camera of a UAV during an exploration inspection of the structure by the UAV. Alternatively, obtaining the 3D map of the structure can include importing the 3D map (e.g., via a CAD or like system) from an external source or generating the 3D map using data obtained from an external source.

[0163] At 1104, pre-existing semantic data associated with the structure is loaded into memory (e.g., of a UAV used to perform the technique 1100). The pre-existing semantic data is semantic data that existed for a given structure prior to the initiation of the technique 1100 for the inspection of that structure. The semantic data includes information usable to semantically identify components of the structure. Non-limiting examples of the semantic data may include site plans, development plans, CAD files, or the like. The pre-existing semantic data may be obtained from a server or other device in communication with the UAV.

[0164] At 1106, the 3D map of the structure is aligned with the semantic data. Aligning the 3D map with the semantic data includes comparing dimensions, identified components, and like qualities of the 3D map to determine likenesses and differences. Components or dimensions that are different between the 3D map and the semantic data are identified to update the 3D map according to the semantic data.

[0165] At 1108, component labels are generated for the aligned 3D map. The component labels are or otherwise include or represent data associated with structure components detected via the computer vision process. For example, a component label may include a name or other identifier for a component or other information usable to identify the component. Generating the component labels for the aligned 3D map can include obtaining user input (e.g., from a user device in communication with the UAV) indicating the content of a label to generate for a content or using a machine learning model (e.g., trained based on historical structure inspection data) to generate such content for a label.

[0166] At 1110, component detection is performed according to the component labels. Performing the component detection according to the component labels includes an Al engine (e.g., a classification -based machine learning model) determine components to which the component labels apply. For example, where the user input indicating content of a label to generate is received to update the name of a component on a bounding box of that component, the Al engine can automatically detect other instances of that object type and accordingly update labels for those corresponding components. The labeled components maythen be output as determined components.

[0167] FIG. 12 is a block diagram of an example of a semantic segmentation rendering system 1200. The system 1200 includes a UAV 1202 and a structure 1204 to be inspected using the UAV 1202. The UAV 1202 may, for example, be the UAV 102 shown in FIG. 1, the UAV 200 shown in FIGS. 2A-B, the UAV 500 shown in FIG. 5, and / or the UAV 602 shown in FIG. 6. The structure 1204 is a physical, 3D structural object that can be wholly or partially flown around by the UAV 1202. Non-limiting examples of the structure 1204 include buildings (e.g., residential, commercial, or industrial, including internal portions (e.g., warehouse aisles)), bridges (e.g., utility, walkway, or roadway), towers (e.g., powerline transmission, radio, cellular, or water), energy equipment (e.g., sub-stations, wind turbines, solar cells or farms, or power plant cooling towers), vehicles (e.g., aircraft, marine vessels, spacecraft, or land vehicles), civil spaces (e.g., parks or parking lots), and geographic features (e.g., forests, bluffs, mountains, valleys, or ravines). In some cases, the structure 1204 may include a living being rather than or in addition to an object as described above. Non-limiting examples of the living being in such a case include humans, animals, fungi, and plants.

[0168] The UAV 1202 includes hardware and software that configure the UAV 1202 to determine a semantic understanding and perform semantic segmentation for a rendering of the structure 1204 via an exploration inspection thereof. In particular, and in addition to other components as are described with respect to FIG. 5, the UAV 1202 includes semantic segmentation and rendering software 1206 and one or more cameras 1208. The semantic segmentation and rendering software 1206, which will be described further in connection with FIG. 13, includes functionality for enabling the UAV 1202 to perform an exploration inspection of the structure 1204 to capture one or more images of the structure 1204 using the one or more cameras 1208, encode identifiers associated with components 1210 of the structure 1204 within pixel values of the one or more images, segment the pixel values into polygons according to the encoded identifiers, and communicate (e.g., to a user device 1212, via a user interface 1214, or to a server 1216) semantic output associated with the components 1210 for use in a further inspection of the components 1210.

[0169] The components 1210 generally are, include, or otherwise refer to components (e.g., objects, elements, pieces, equipment, sub-equipment, tools, or other physical matter) on or within the structure 1204. In one non-limiting example, where the structure 1204 is a powerline transmission tower, the components 1210 may include one or more of an insulator, a static line or connection point, a conductor or overhead wire, a footer, or a transformer). The structure 1204 may include any number of types of the components 1210 and anynumber of ones of the components 1210 for each of the individual types thereof.

[0170] The user device 1212 is a computing device configured to communicate with the UAV 1202 wirelessly or by wire. For example, the user device 1212 may be one or more of the controller 104 shown in FIG. 1, the controller 300 shown in FIG. 3, or the user device 612 shown in FIG. 6. In some cases, the UAV performs the exploration inspection of the structure 1204 using input obtained from the user device 1212. For example, the input obtained from the user device 1212 may include information specifying one or more of the components 1210 to inspect, a height at which the UAV 1202 will fly while traversing along a flight path, a distance for the UAV 1202 to fly, a maximum speed for the UAV 1202 to fly, a radius that the UAV 1202 will use to orbit or otherwise explore and evaluate the structure 1204, a gimbal angle for the one or more cameras 1208, a column path for the flight path, an instruction for data capture at waypoints of the flight path, or an instruction for stopping the inspection of the structure 1204.

[0171] In some cases, the user of the user device 1212 may specify one or more configurations for controlling the capture of some or all of the one or more images during the exploration inspection. For example, user input may be obtained from the user device 1212 (e.g., via the user interface 1214, which may, for example, include one or more graphical user interfaces (GUIs) for input collection) to configure the UAV to capture a single image for each component of the structure, to capture N images of the structure each from a different manually-specified or automatically-determined angle (i.e., where N is an integer greater than 1), to capture images from a specified ground sampling distance (GSD) (e.g., using the same GSD for the entire exploration inspection or using different GSDs for different portions of the structure), or the like.

[0172] The UAV 1202, via the semantic segmentation and rendering software 1206, may utilize empirical and / or machine learning-based data modeled for use in structure inspections. In particular, the UAV 1202 may communicate with the server 1216, which includes a data library 1218 usable by or otherwise in connection with the semantic segmentation and rendering software 1206 to store semantic output generated using the semantic segmentation and rendering software 1206 and / or to serve, according to such stored semantic output, responses to queries for images that depict the structure 1204 or one or more of the components 1210. The server 1216 is a computing device remotely accessible by or otherwise to the UAV 1202. The data library 1218 may, for example, include one or more of semantic output generated for the structure 1204 and / or other structures, historic inspection data for the structure 1204 and / or the other structures, machine learning models (e.g.,classification engines comprising trained convolutional or deep neural networks) trained according to inspection image output data sets with user-specific information culled, and / or other information usable by the system 1200. In some cases, the data library 1218 or other aspects at the server 1216 may be accessed by or otherwise using the user device 1212 instead of the UAV 1202.

[0173] To further describe functionality of the semantic segmentation and rendering software 1206, reference is next made to FIG. 13, which is a block diagram of example functionality of the semantic segmentation and rendering software 1206. The semantic segmentation and rendering software 1206, as described above, may use various functionality of a UAV (e.g., the UAV 1202 shown in FIG. 12) to cause the UAV to generate semantic output based on encoded identifiers of pixel values of images captured for a site including one or more structures, for example, the structure 1204 shown in FIG. 12. The semantic segmentation and rendering software 1206 includes a graphical representation processing tool 1300, an identifier encoding tool 1302, a pixel value segmentation tool 1304, and a semantic output processing tool 1306. The tools 700 through 706 represent various functionality of the semantic segmentation and rendering software 1206 and are non-limiting as to a particular structure or other expression of code, script, or the like. While the semantic segmentation and rendering software 1206 may be used to sequentially or simultaneously process semantic output for multiple structures at a site, the semantic segmentation and rendering software 1206 will be described with respect to processing semantic output for a single structure.

[0174] The graphical representation processing tool 1300 obtains a 3D graphical representation of a structure under inspection. The 3D graphical representation is a visual model of the structure and may, for example, be a parametric 3D model. The graphical representation processing tool 1300 obtains the 3D graphical representation of the structure either by generating the 3D graphical representation or by importing the 3D graphical representation, for example, via a computer-aided design (CAD) system. Where the graphical representation processing tool 1300 generates the 3D graphical representation of the structure, the graphical representation processing tool 1300 does so using, as input, one or more images of the structure captured using a UAV while the UAV orbits around some or all of the structure. Orbiting around the structure may, for example, include the UAV circling the structure, navigating along a perimeter or like boundary of the structure, or otherwise exploring the structure from a distance. In some cases, the orbiting can follow a zig-zag, lawnmower, or other pattern around the structure. The orbiting may be performed at one or more altitudes relative to the structure.

[0175] The graphical representation processing tool 1300 generates the 3D graphical representation of the structure based on the one or more images using a ray -based optimization technique. The ray-based optimization technique is a technique for using rays to determine where information passes through the one or more images on a per pixel basis. In particular, a ray as used by such a ray -based optimization technique is a linear projection that intersects an image at a single pixel. Each pixel is associated with a different ray. The rays are grouped together to derive understandings of objects according to their shapes, sizes, and relative locations. For example, the ray -based optimization technique may be a graph clustering technique that measures the relative scores of clusters of rays, in which each ray represents or otherwise corresponds to a node of a graph. In particular, with a graph clustering technique, distances are measured each between a given pair of rays in physical space. Those distances are modeled using empirical understandings via semantic segmentation modeling and / or machine learning model-based (e.g., deep learning-based) classifications to infer a component type for an object depicted in an image of the one or more images.

[0176] In addition to the pixel content of the one or more images, the graphical representation processing tool 1300 obtains location information for the various objects detected within the one or more images via the ray-based optimization described above. For example, the location information may be expressed in 3D space using XYZ coordinates and determined using pose information of the UAV to capture the one or more images alongside one or more of a visual positioning system (VPS) of the UAV used, GPS, or another locationbased system. The location information is used by the ray-based optimization technique to recognize same objects depicted in different images of the one or more images, noting that such depictions of a same object may vary according to a given pose taken by the UAV at the time of image capture. For example, the location information may be determined using the VPS, GPS, or other location-based system via a triangulation process performed at the UAV.

[0177] The output of the graphical representation processing tool 1300 thus includes a 3D graphical representation of the structure, identifications within the 3D graphical representation of components of the structure, and location information for the structure and its components. In particular, the result of the ray -based optimization technique includes a set of points in 3D space that may be represented as a 3D model such as a point cloud, which may be fitted with 3D a parametric model to finetune parameters of the 3D model, to result in the 3D graphical representation of the structure. The 3D graphical representation thus operates in place of bounding boxes, cuboids, or the like. In some cases, finetuning theparameters of the 3D model can include detecting key points in the images of the structure and matching those detected key points with the respective key points of the 3D model by running a genetic algorithm that updates the parameters of the 3D model. Non-limiting examples of parameters which may be finetuned in this way include dimensions of individual components (e.g., height, width, and / or depth), twists or bends and tolerances therefor for individual components, rotations (e.g., pitch, roll, and / or yaw) for individual components, and the like.

[0178] The identifier encoding tool 1302 processes the one or more images of the structure that were captured using the UAV to encode one or more identifiers in some or all pixel values of some or all of those one or more images. In particular, in each pixel value that corresponds to the structure within the one or more images, identifiers of the structure, of a component of the structure depicted using the pixel value, and of a location of the component are encoded. In some cases, some, but not all of those identifiers may be encoded in the respective pixel values. In some cases, additional identifiers may be encoded in respective pixel values. For example, the additional identifiers may indicate context of a component depicted by a respective pixel value, such as whether the pixel value is of a top or bottom surface of the component Encoding the identifiers in a pixel value includes generating metadata indicative of the identifiers and associating that metadata with the pixel value.

[0179] The identifier encoding tool 1302 encodes the identifiers in the respective pixel values as part of a process for rendering the 3D graphical representation of the structure with shaders for visually distinguishing between components of the structure. Thus, simultaneous or sequential processing may occur between the identifier encoding tool 1302 and the graphical representation processing tool 1300. In particular, for each image of the structure, the graphical representation processing tool 1300, using a 3D engine, renders one or more portions of the 3D graphical representation, each portion corresponding to one or more components of the structure, with respective shaders. In one non-limiting example, the shaders may each be or otherwise correspond to different colors or shading styles. For example, the 3D engine may render the one or more portions of the 3D graphical representation for a given image by first encoding an identifier of the structure in some or all pixel values of the image and thereafter encoding other identifiers in those pixel values. For example, the other identifiers may include component identifiers, location identifiers, or other identifiers.

[0180] Rendering the 3D graphical representation of the structure includes generating a UV map of the structure according to the shaders used for the respective components of thestructure and then rendering the 3D graphical representation of the structure based on the UV map. The UV map is a two-dimensional (2D) representation of the pixel values depicting the structure from the images captured using the UAV In particular, the UV map identifies an arrangement of the components of the structure based on shaders used for those components. The 3D graphical representation of the structure may be rendered based on the UV map by projecting the UV map onto a surface of the 3D graphical representation, in which different indices, or points, of the UV map are related to different locations on the 3D graphical representation. Thus, the identifier encoding tool 1302 may encode, as the other identifiers in respective pixel values, UV map indices of those pixel values. Ultimately, the encoding of the identifiers in the pixel values enables those pixel values to indicate semantic information of the structure and its components.

[0181] The pixel value segmentation tool 1304 segments the pixel values of the images into polygons of the 3D graphical representation of the structure according to the identifiers encoded in the pixel values using the identifier encoding tool 1302. Segmenting the pixel values into the polygons includes performing a computer vision process to segment the data of the pixel values into vector annotations representing the polygons. For example, the computer vision process may be or include panoptic segmentation performed to identify classes of pixel content while distinguishing between instances of such classes elsewhere in the image data. Thus, individual components of the same type of the structure may be separately identified and accordingly distinguished using panoptic segmentation, making it a valuable computer vision process for uniquely identifying structure components by their types and locations. The pixel value segmentation tool 1304 accordingly operates to separately identify components by polygon based on their encoded identifiers.

[0182] The semantic output processing tool 1306 processes the output of the semantic segmentation and rendering software 1206. In particular, the semantic output processing tool 1306 operates to cause a storage of data indicative of the polygons in a data store for use in a further inspection of the structure at a same or later date as when the pixel values are segmented into the polygons. In some cases, the data indicative of the polygons identifies respective ones of the shaders used for the components of the structure. For example, the data indicative of the polygons may include the UV map generated for the structure. The data store in which the data indicative of the polygons is stored may, for example, be a server-side (i.e., cloud-based) data store accessible by the UAV and / or the user device and one or more other UAVs and / or user devices in communication therewith. Alternatively, or additionally, the semantic output processing tool 1306 may operate to cause the user device incommunication with the UAV to output data associated with the 3D graphical representation of the structure at the user device.

[0183] The semantic segmentation and rendering software 1206 is shown and described as being run (e.g., executed, interpreted, or the like) at a UAV used to perform an inspection of a structure. However, in some cases, the semantic segmentation and rendering software 1206 or one or more of the tools 700 through 706 may be run other than at the UAV. For example, one or more of the tools 700 through 706 may be run at a user device or a server device and the output thereof may be communicated to the UAV for processing.

[0184] In some implementations, the UV map generated as part of the rendering of the 3D graphical representation of the structure using the graphical representation processing tool 1300 can be used to determine a flight path for inspecting the structure according to relative locations of the components represented within the indices of the UV map. For example, the UAV, via the semantic segmentation and rendering software (e.g., via the semantic output processing tool 1306 or another tool) may automatically determine the flight path and present it on the UV map as an overlay. In another example, the user of a user device in wired or wireless communication with the UAV may, via a GUI that outputs the UV map for display at the user device, trace or otherwise indicate the flight path for the structure directly on the UV map.

[0185] FIG. 14 is an illustration of an example of a structure 1400 and its components for which semantic output may be generated. In particular, the structure 1400 in the example shown is a wind turbine. The wind turbine includes several components identified by the graphical representation processing of images captured using a UAV, such as described above with respect to the semantic segmentation and rendering software 1206. As shown, the structure 1400 includes a tower 1402, a nacelle 1404, a rotor and hub 1406, and three blade assemblies, each including a leading edge (1408A, 1408B, 1408C), a trailing edge (1410A, 1410B, 1410C), and a tip (1412A, 1412B, 1412C). Images of the structure 1400 are processed as described with respect to FIG. 13 to identify each of the components 1402 through 1412A-C and to encode identifiers in pixel values depicting ones of the components 1402 through 1412A-C. This enables the rendering of those components 1402 through 1412A-C in a 3D graphical representation of the structure 1400 using different shaders to visually distinguish between the components 1402 through 1412A-C. For example, unique shaders may be used for each individual component of the structure 1400 such that no two components 1402 through 1412A-C share a same shader. This includes components of a same type - thus, representations of the leading edges 808A, 808B, and 808C within the 3Dgraphical representation of the structure 1400 are rendered using three different shaders.

[0186] FIG. 15 is an illustration of an example of a UV map 1500 for a structure and encodings 1502 for a pixel value corresponding to a portion of the structure. The UV map is a two-dimensional representation of surface content of a 3D graphical representation of a structure, in this case, the structure 1400 shown in FIG. 14. The colors shown and the locations of those colors in the UV map 1500 correspond to the shaders and the components on which the shaders are used of the structure 1400. Noting that the structure 1400 is depicted in FIG. 14 in black and white, the UV map 1500 simply demonstrates one example of how the shader contents may appear in a UV map. Moreover, the particular arrangement of shaders in the UV map 1500 is, as described above, based on the locations of the components that use those shaders, and so the arrangement of the colors illustrated in the UV map 1500 is by example only. Separately, the encodings 1502 represent an example visualization of the encoded identifiers for a pixel value of the structure 1400, which pixel value is located at coordinates (U, V) of the UV map 1500. As shown, the encodings 1502 indicate a structure identifier for the pixel value, a component identifier for the pixel value, and XYZ coordinates of the pixel value in some 3D space range, an index of the pixel value within the UV map 1500.

[0187] To further describe some implementations in greater detail, reference is next made to examples of techniques for semantic segmentation rendering. FIG. 16 is a flowchart of an example of a technique 1600 for semantic segmentation rendering for precise localization of structure components on captured images. The technique 1600 can be executed using computing devices, such as the systems, hardware, and software described with respect to FIGS. 1-5 and 12-15. The technique 1600 can be performed, for example, by executing a machine-readable program or other computer-executable instructions, such as routines, instructions, programs, or other code. The steps, or operations, of the technique 1600 or another technique, method, process, or algorithm described in connection with the implementations disclosed herein can be implemented directly in hardware, firmware, software executed by hardware, circuitry, or a combination thereof.

[0188] For simplicity of explanation, the technique 1600 is depicted and described herein as a series of steps or operations. However, the steps or operations in accordance with this disclosure can occur in various orders and / or concurrently. Additionally, other steps or operations not presented and described herein may be used. Furthermore, not all illustrated steps or operations may be required to implement a technique in accordance with the disclosed subject matter.

[0189] At 1602, one or more images of a structure are obtained. The one or more images are captured using a UAV during an exploration inspection of the structure. In some cases, the one or more images may be of multiple structures at a site.

[0190] At 1604, a 3D graphical representation of the structure is obtained and rendered by encoding identifiers in pixel values of the one or more images. In particular, in each pixel value that corresponds to the structure within the images, identifiers are encoded. The identifiers include, for example, identifiers of the structure, of components depicted by the respective pixel value, of locations of the components, and the like. The location of a component may, for example, be determined using a VPS of the UAV and pose information of the UAV (e.g., indicating a pose the UAV took while capturing an image of the component). In some cases, the identifiers encoded in a pixel value may include an index of a UV map at which the pixel value corresponds. Obtaining the 3D graphical representation of the structure may include importing the 3D graphical representation as a 3D model (e.g., from a CAD file) or generating the 3D graphical representation based on the images captured using the UAV. For example, generating the 3D graphical representation can include generating same using a ray -based optimization technique, such as graph clustering.Rendering the 3D graphical representation includes rendering the 3D graphical representation using shaders to visually distinguish each component of the structure and can include generating the UV map of the structure according to shaders used with the components of the structure and rendering the 3D graphical representation based on the UV map. In some cases, rendering the 3D graphical representation can include rendering different surfaces of a component of the structure within the 3D graphical representation using different ones of the shaders. For example, top and bottom surfaces of a single component may be separately rendered using different colors as the different shaders.

[0191] At 1606, the pixel values of the images are segmented into polygons according to the encoded identifiers of those pixel values. Segmenting the pixel values of the images into the polygons according to the encoded identifiers of those pixel values can include performing a computer vision process (e.g., panoptic segmentation) against the images to segment the pixel values into vector annotations corresponding to the polygons. The polygons correspond to ones of the components of the structure and thus the 3D graphical representation of the structure is segmented to separately identify the components.

[0192] At 1608, data indicative of the polygons is stored. For example, the data indicative of the polygons may include the UV map, the encoded identifiers of the pixel values, and / or other data that identifies respective ones of the shaders used for respective components of thestructure. The data indicative of the polygons is stored in a data store, such as a server-side data store accessible by multiple UAVs and / or communicating devices. The data indicative of the polygons is stored for one or more purposes, including, for example, for use in a further inspection of the structure.

[0193] At 1610, the data indicative of the polygons is used to indicate images which include structure based on a query. For example, a query for images which depict the structure may be received from the UAV, the user device in communication with the UAV, or another UAV, user device, or computing device. The query is processed to generate a query response that indicates the images which depict the structure.

[0194] FIG. 17 is a block diagram of an example of a multi -phase semantic 3D scan system 1700. The system 1700 includes a UAV 1702 and a structure 1704 to be inspected using the UAV 1702. The UAV 1702 may, for example, be the UAV 102 shown in FIG. 1, the UAV 200 shown in FIGS. 2A-B, the UAV 500 shown in FIG. 5, the UAV 602 shown in FIG. 6, and / or the UAV 1202 shown in FIG. 12. The structure 1704 is a physical, 3D structural object that can be wholly or partially flown around by the UAV 1702. Non-limiting examples of the structure 1704 include buildings (e.g., residential, commercial, or industrial, including internal portions (e.g., warehouse aisles)), bridges (e.g., utility, walkway, or roadway), towers (e.g., powerline transmission, radio, cellular, or water), energy equipment (e.g., sub-stations, wind turbines, solar cells or farms, or power plant cooling towers), vehicles (e.g., aircraft, marine vessels, spacecraft, or land vehicles), civil spaces (e.g., parks or parking lots), and geographic features (e.g., forests, bluffs, mountains, valleys, or ravines). In some cases, the structure 1704 may include a living being rather than or in addition to an object as described above. Non-limiting examples of the living being in such a case include humans, animals, fungi, and plants.

[0195] The UAV 1702 includes hardware and software that configure the UAV 1702 to perform a multi-phase semantic 3D scan of the structure 1704. In particular, and in addition to other components as are described with respect to FIG. 5, the UAV 1702 includes multiphase semantic 3D scan software 1706 and one or more cameras 1708. The multi-phase semantic 3D scan software 1706, which will be described further in connection with FIG. 18, includes functionality for enabling the UAV 1702 to perform a multi-phase inspection of the structure 1704 as a semantic 3D scan including a first phase inspection via which the UAV 1702, using the one or more cameras 1708, captures images used to determine components 1710 of the structure 1704 and a second phase inspection via which the UAV 1702, using the one or more cameras 1708, captures images of a subset of the components 1710 according toa flight path for the semantic 3D scan.

[0196] The components 1710 generally are, include, or otherwise refer to components (e.g., objects, elements, pieces, equipment, sub-equipment, tools, or other physical matter) associated with (i.e., on or within) the structure 1704. In one non-limiting example, where the structure 1704 is a powerline transmission tower, the components 1710 may include one or more of an insulator, a static line or connection point, a conductor or overhead wire, a footer, or a transformer). The structure 1704 may include any number of types of the components 1710 and any number of ones of the components 1710 for each of the individual types thereof.

[0197] The output of the multi-phase semantic 3D scan inspection of the structure 1704 includes at least one or more images, captured using the one or more cameras 1708, of a subset of the components 1710. The output may be communicated to a user device 1712 via a user / UAV interface 1714. The user device 1712 is a computing device configured to communicate with the UAV 1702 wirelessly or by wire. For example, the user device 1712 may be one or more of the controller 104 shown in FIG. 1, the controller 300 shown in FIG.3, the user device 1712 shown in FIG. 17, or the user device 1212 shown in FIG. 12. In some cases, the UAV performs the multi-phase semantic 3D scan inspection of the structure 1704 using input obtained from the user device 1712. For example, the input obtained from the user device 1712 may include information specifying one or more of the components 1710 to inspect, a height at which the UAV 1702 will fly while traversing along a flight path, a distance for the UAV 1702 to fly, a maximum speed for the UAV 1702 to fly, a radius that the UAV 1702 will use to orbit or otherwise explore and evaluate the structure 1704, a gimbal angle for the one or more cameras 1708, a column path for the flight path, an instruction for data capture at waypoints of the flight path, or an instruction for stopping the inspection of the structure 1704.

[0198] The UAV 1702, via the multi-phase semantic 3D scan software 1706, may utilize empirical and / or machine learning-based data modeled for use in structure inspections. In particular, the UAV 1702 may communicate with a server 1716 that includes a data library 1718 usable by the multi -phase semantic 3D scan software 1706 to perform a multi-phase semantic 3D scan inspection of the structure 1704. The server 1716 is a computing device remotely accessible by or otherwise to the UAV 1702. The data library 1718 may, for example, include one or more of historic inspection data for the structure 1704 or like structures, machine learning models (e.g., classification engines comprising trained convolutional or deep neural networks) trained according to inspection image output data setswith user-specific information culled, and / or other information usable by the system 1700. In some cases, the data library 1718 or other aspects at the server 1716 may be accessed by or otherwise using the user device 1712 instead of the UAV 1702.

[0199] To further describe functionality of the multi-phase semantic 3D scan software 1706, reference is next made to FIG. 18, which is a block diagram of example functionality of the multi-phase semantic 3D scan software 1706. The multi-phase semantic 3D scan software 1706, as described above, may use various functionality of a UAV (e.g., the UAV 1702 shown in FIG. 17) to cause the UAV to perform a multi -phase semantic 3D scan for the inspection of a structure. The multi-phase semantic 3D scan software 1706 includes a user input processing tool 1800, a structure component determination tool 1802, a flight path determination tool 1804, and a semantic 3D scan inspection tool 1806. The tools 700 through 706 represent various functionality of the multi-phase semantic 3D scan software 700 and are non-limiting as to a particular structure or other expression of code, script, or the like.

[0200] The user input processing tool 1800 obtains and processes input obtained from an operator of the UAV for the multi-phase semantic 3D scan of the structure (e.g., the structure 1704 shown in FIG. 17). The input includes configuration information for the scan and may be obtained in one or more formats, including free-text, selections from listed options, or the like. For example, the input may include information specifying one or more of objects of interest for the scan (e.g., the components to inspect, such as the components 1710 shown in FIG. 17), a height (i.e., altitude) at which the UAV will fly while traversing along a flight path (e.g., between columns of the flight path, as described below), a distance for the UAV to fly from a boundary of the structure, a maximum speed for the UAV to fly (e.g., in miles per hour or kilometers per hour), a radius that the UAV will use to orbit or otherwise explore and evaluate the structure during a first phase inspection of the structure (e.g., in which the radius is larger than the sum of a stand-off distance and half of a dimensional measurement associated with the structure), a gimbal angle for one or more cameras of the UAV (e.g., set for individual objects or universally for all objects, in which the gimbal angle may by default be zero and in which multiple gimbal angles may accordingly be set), a column path for the flight path (e.g., indicating whether to use two or four columns), an instruction for data capture at waypoints of the flight path (e.g., enabling or disabling the automatic image capture at each waypoint of the flight path during autonomous flight), or an instruction for stopping the inspection of the structure (e.g., enabling or disabling the UAV pausing at each waypoint of the flight path during autonomous flight). The user input processing tool 1800 processes this input and passes same for use with other tools of the multi-phase semantic 3Dscan software 1706 to configure the multi-phase inspection of the structure according to the input.

[0201] As will be described in more detail below, additional user input may be obtained and processed using the user input processing tool 1800 following a first inspection phase of the multi-phase semantic 3D scan of the structure. In particular, in some cases, output of the first inspection phase indicating a semantic understanding of components associated with (i.e., on or within) the structure and pose information of the components may be presented to an operator of the UAV (e.g., via the user device or a secondary device in communication with the user device or otherwise registered for use with the UAV). The output of the first inspection phase in such cases is presented to enable the operator of the UAV to indicate ones of the components which they would like the UAV to inspect during a second inspection phase of the multi-phase semantic 3D scan of the structure. For example, a list of components determined via the first inspection phase may be presented within a graphical user interface (GUI) output for display at the user device or a secondary device. The list may include individual entries each corresponding to a different component type. For example, where the structure is a powerline transmission tower, the GUI may include a first entry for insulators, a second entry for connection points, a third entry for conductors, and so on. The entries may each have a selectable or otherwise interactive element (e.g., a checkbox). When such an interactive element is selected or otherwise interacted, the additional user input is obtained, in which the additional user input indicates an intention of the UAV operator to cause the respective components associated with the subject entry to be inspected during a second phase inspection of the multi-phase semantic 3D scan of the structure. However, selections of components in this manner may instead be performed automatically by the UAV or otherwise by a computing device in communication with the UAV (e.g., using intelligence of detected components independently, relative to the structure, or in view of an inspection purpose for the structure (e.g., known issues resulting in the performance of the multi-phase semantic 3D scan of the structure)). Thus, the additional user input as described herein may be limited to manual processes for indicating components rather than automated approaches for indicating components.

[0202] In some cases, the additional user input may be provided partially before and partially after performance of the second-phase inspection. For example, after the second- phase inspection begins, the user input processing tool 1800 may receive, via a GUI output for display at the user device or a secondary device, input indicating control values for capturing the one or more images during the second phase inspection. The control values aregenerally values usable by the UAV and / or the multi-phase semantic 3D scan software 1706 to control some aspect of the second phase inspection and may, in non-limiting examples, refer to one or more of gimbal angles for the UAV camera(s), viewpoints of the UAV relative to components to inspect, or incline values indicating how the UAV is to approach the component in one or more directions.

[0203] The structure component determination tool 1802 determines semantic understandings of components associated with (i.e., on or within) the structure to inspect and pose information of the components. As will be described in more detail below, a first inspection phase of the multi-phase semantic 3D scan of the structure includes the UAV capturing one or more images (i.e., exploration images) of the structure while navigating about an exploration path around some or all of the structure. The structure component determination tool 1802 obtains, as input, those one or more images captured during the first inspection phase and processes them to determine, as output, semantic understandings of the components to inspect and the pose information of the components. In particular, the structure component determination tool 1802 processes an image captured by the UAV during the first inspection phase to detect one or more objects, as one or more components, within the image and to determine pose information of those one or more components using navigation system information of the UAV.

[0204] Detecting the one or more objects within the image as the one or more components includes performing object detection against the image to identify a bounding box for each component within the image. Objects detected within the image are thus represented via their bounding box, and object recognition is performed against the bounded objects to identify them as components of the structure, as well as to identify what components they are. In some cases, performing object recognition includes comparing objects detected within the images against modeled object data to determine whether an object appears as expected. The objects detected and recognized are identified as components and information indicating the components is stored in connection with the bounding boxes of the components. In some cases, image segmentation may be performed instead of object detection and object recognition. In some cases, other computer vision techniques may be performed instead of either of the above approaches.

[0205] Determining the pose information of the one or more components detected within the image using the navigation system information of the UAV includes determining orientations and / or locations of the detected components based on their bounding boxes. The pose information of the components thus corresponds to or otherwise represents theorientations and / or locations of the different components within a scene of the structure. In particular, the pose information of a given component may identify sides, facets, surfaces, or like qualities of the component independently and with relation to others. For example, the pose information may identify a front of a component such that a location of the front of the component may be identifiable. In some cases, the pose information of a component may be based on the type of the component. For example, because inspectors of powerline transmission towers generally need to view insulators from their top and bottom sides, the pose information for an insulator component may include a top pose above the insulator and a bottom pose below the insulator.

[0206] Because it is important that objects be uniquely identified to prevent multiple of the same component from being confused as being the same exact component, the structure component determination tool 1802, as part of the component and pose information determination, performs a triangulation process to ensure that specific component location information is known. In particular, locations of bounding boxes for detected components may be denoted using location data according to one or more of a visual positioning system (VPS) of the UAV, GPS, or another location-based system. The location data for bounding boxes of the detected components may be compared to determine duplicate components, in which a duplicate component is a component that has already been detected in a previous image. Duplicate components are accordingly culled to prevent them from being considered for scanning multiple times.

[0207] The flight path determination tool 1804 determines a flight path for the UAV to navigate within the scene of the structure based on the components and the pose information of the components, as determined using the structure component determination tool 1802. In particular, the flight path includes capture points (e.g., waypoints) to which the UAV will navigate for capturing one or more images of individual components of the components during the second phase inspection of the structure. The capture points are expressed as a sequence within the flight path according to a trajectory determined using the flight path determination tool 1804. The capture points are generally arranged into a number of columns, in which a column generally refers to a vertical airspace through which the UAV will ascend, descend, and / or otherwise navigate to capture one or more images of one or more components visible from that column according to the respective pose information. The specific size of a column is non-limiting, however generally a height of a column will exceed a width of the column. The flight path thus causes the UAV to navigate between all columns during the second phase inspection.

[0208] Determining the flight path for the UAV to navigate includes determining a trajectory for the UAV to navigate proximate to locations of the bounding boxes determined using the structure component determination tool 1802. Determining the trajectory includes determining a number of columns for the UAV to vertically navigate based on the locations of the components according to location information identified for their respective bounding boxes, in which each column of the number of columns includes one or more of the capture points. In particular, a bounding box determined for a component may identify a VPS-based, GPS-based, or other location for the component within the scene of the structure. The location of a bounding box may be recorded within data or metadata associated with the bounding box. Thus, the flight path determination tool 1804 obtains the location information for each of the bounding boxes (i.e., for each of the components) and determines the trajectory accordingly.

[0209] The trajectory generally refers to a manner by which the UAV may fly to visually access each of the components to inspect during the second phase inspection. The flight path determination tool 1804 uses the pose information for the components to determine locations at which the UAV may visually access respective ones of the components. Visual access as used herein generally refers to the UAV navigating to a location about a structure at which a subject component may be positioned entirely within a field of view of a camera of the UAV. The visual access may be based on specific pose information for the subject component, for example, to ensure that a specific facet of the component is visible in which the location of that facet is known based on the pose information of the component.

[0210] To ensure inspection efficiency, however, recognizing that pose information may express visual access for a given component at multiple possible locations, determining the flight path includes determining a number of columns for the UAV to vertically navigate. The number of columns are determined to wholly or partially surround some or all of the structure. For example, the number of columns may be two or four. The particular number of columns to use for a given inspection may be dependent upon the location of the structure relative to obstructions or other structures in the scene, the type of the structure, and / or other factors. The use of columns for vertical navigation restricts horizontal traversal of the UAV proximate to the structure for the safety and security of the UAV and the structure.

[0211] The flight path determination tool 1804 determines the number of columns according to the trajectory and thus the locations of the components and the pose information thereof. For example, flight path mapping software as is commonly used in UAV inspection systems may be used to determine an optimal trajectory according to the location and poseinformation of the components. The number of columns may thus be automatically determined based on the output of such flight path mapping functionality. In some cases, the number of columns may be determined based on user input processed using the user input processing tool 1800. In either case, the locations of the columns are also determined according to the number of columns. For example, where two columns may be used to visually access all necessary components to inspect during the second phase inspection, the two columns may be located at opposing diagonal ends of the structure. In another example, where four columns may be used to visually access all such components, the four columns may form a rectangular boundary around the relevant part or whole of the structure and thus be located at corners of that rectangular boundary.

[0212] The flight path determination tool 1804 then uses the number of columns and the trajectory to determine the flight path for the second phase inspection of the structure. In particular, the flight path generally corresponds to the trajectory but aligns with the locations of the columns to ensure that the UAV efficiently navigates between the columns. The flight path generally will cause the UAV to navigate between the columns at a certain altitude above the structure and then to descend to each of one or more capture point locations associated with the column to enable a capture of images of respective components visibly accessibly at the subject column. The flight path may generally begin at a first column and end at a last column. For example, navigating the flight path may include the UAV flying from an initial location to a location of the first column to begin the flight path.

[0213] The flight path determination tool 1804 further determines camera poses for a camera of the UAV, to correspond with the capture points of the flight path. In particular, a camera pose represents a manner by which to orient a camera of the UAV in order to capture an image of a subject component at a subject capture point. The camera pose data may be embedded within the flight path using VPS, GPS, or another location-based system.

[0214] In some cases, an operator of the UAV, via a GUI output for display at the user device or a secondary device, may alter the flight path after it is determined. For example, the GUI may provide a graphical representation of the flight path about the scene of the structure and may include one or more user interface tools configured to enable a modification of some or all of the flight path. In one non-limiting example, a touch interface or click-and-drag approach may be used to alter one or more lines of the flight path.

[0215] In some cases, the flight path determination tool 1804 may initiate the determination of the flight path using a template associated with the structure of a type of the structure. For example, the template may include capture points in a number of columns thatare most commonly used for the specific structure under inspection or for structures of the same type as that specific structure. In some such cases, the operator of the UAV may modify some or all of the templatized flight path in the manner described above.

[0216] While the above describes the functionality of the flight path determination tool 1804 with respect to all components determined by the structure component determination tool 1802, in some cases, the flight path determination tool 1804 determines a flight path for only a subset of those components. For example, the user input processed by the user input processing tool 1800 may specify only certain types of components or only certain components to inspect. The flight path determination tool 1804 may thus reference the components determined using the structure component determination tool 1802 against the components represented within that user input to determine a subset of the components for which to determine the flight path. In some such cases, the subset of components for which to determine the flight path may also or instead be determined based on additional user input obtained using the user input processing tool 1800 (e.g., after the first inspection phase of the multi-phase semantic 3D scan of the structure). In another example, an automated determination of a subset of the components determined using the component determination tool 1802 may be made based on historical scan data for the structure or similar structures, information related to the structure or similar structures, and / or like factors.

[0217] The semantic 3D scan inspection tool 1806 causes a performance of a multi -phase inspection of the structure, including a first phase inspection performed to determine a semantic understanding of the components and the pose information of the components and a second phase inspection performed according to the flight path to determine output associated with some or all of those components. In particular, to cause a performance of the first phase inspection, the semantic 3D scan inspection tool 1806 causes a navigation system of the UAV (e.g., via the propulsion control interface 510 shown in FIG. 5) to fly the UAV so as to orbit around the scene that includes the structure. Orbiting around the scene may, for example, include the UAV circling the structure, navigating along a perimeter or like boundary of the structure, or otherwise exploring the scene of the structure from a distance. In some cases, the orbiting can follow a zig-zag, lawnmower, or other pattern around the scene of the structure. The orbiting may be performed at one or more altitudes relative to the structure.

[0218] To begin the first phase inspection, a location of the structure is provided via VPS, GPS, or a like location service. The semantic 3D scan inspection tool 1806 instructs the navigation system of the UAV to navigate toward the location of the structure at an altitude above the structure while instructing one or more cameras of the UAV to capture imagesand / or video of the structure with the structure visible and centered within a field of view of the one or more cameras. The UAV accordingly navigates toward the structure with the entire structure visible and centered within the field of view of the one or more cameras until the structure is large enough to cover a threshold percentage (e.g., 50 percent) of the images or video captured by the one or more cameras (a threshold percentage of the pixels of the images or video frames). Based on a determination that the structure covers the threshold percentage of the images or video, the UAV begins its orbit around the structure while continuing to capture images or video (i.e., exploration images) until all portions of the structure have been captured within at least one image or video frame. Those images or video captured while the UAV orbits around the structure are then provided to the structure component determination tool 1802 for use in determining the components of the structure and pose information of the components.

[0219] To cause a performance of the second phase inspection, the semantic 3D scan inspection tool 1806 causes the navigation system of the UAV to fly the UAV according to the flight path determined using the flight path determination tool 1804. When the UAV arrives at a given capture point of the flight path, the semantic 3D scan inspection tool 1806 causes the UAV, via the one or more cameras of the UAV, to perform a fine-aiming process by which a camera of the UAV is aimed at a component of the structure according to a camera pose represented within or otherwise by the flight path. The semantic 3D scan inspection tool 1806 determines whether the component is centered within a field of view of the camera (e.g., based on the component covering a threshold percentage (e.g., fifty percent) of the field of view). Based on a determination that the component is centered within the field of view of the camera, the semantic 3D scan inspection tool 1806 causes the camera to capture an image of the component. The UAV then navigates to a next capture point of the flight path and continues to capture images of components and navigate along the flight path until a last capture point is reached and an image is captured thereat. At such a time, the UAV is determined to have completed the flight path and may return to its dock or to another location for further action.

[0220] The output of the second phase inspection includes the images of the components captured at the capture points of the flight path (i.e., inspection images). Those images are labeled with information associated with one or both of the structure or the flight path. For example, the images may be embedded with data or metadata identifying a location of the structure, a name or other identifier associated with the structure, a name or other identifier of respective components depicted within the images, columns of the flight path in which theUAV was located when the images were captured, a date of the inspection, and / or like information.

[0221] The multi-phase semantic 3D scan software 1706 is shown and described as being run (e.g., executed, interpreted, or the like) at a UAV used to perform an inspection of a structure. However, in some cases, the multi -phase semantic 3D scan software 1706 or one or more of the tools 700 through 706 may be run other than at the UAV. For example, one or more of the tools 700 through 706 may be run at a user device or a server device and the output thereof may be communicated to the UAV for processing.

[0222] In some implementations, trajectories other than columns may be used to determine the flight path and to navigate according to the flight path during the inspection phase. For example, the flight path, rather than being based on columns, may instead include an orbit trajectory such as a lawnmower pattern, zig-zag pattern, horizontal line pattern, oblique line pattern, or even random waypoints. The particular type of trajectory may be based on the structure type, and thus the use of columns or the non-use of columns may be based on the structure type. Implementations of this disclosure that address columns may thus be understood to also address alternative trajectories as set forth herein.

[0223] FIG. 19 is a block diagram of an example workflow for multi-phase semantic 3D scan. The workflow generally describes operations performed for the multi-phase semantic 3D scan of a structure, such as using the multi-phase semantic 3D scan software 1706 shown and described with respect to FIGS. 17 and 18, as well as inputs and outputs of those operations. First, user input 1900 is obtained to identify the structure and provide configurations for the multi-phase inspection. A first phase inspection operation 1902 is performed to capture exploration images 1904 based on an orbiting of the UAV about the scene of the structure. The exploration images 1904 are provided to a component and pose information determination operation 1906 to determine components of the structure and pose information of the components 1908. A flight path determination operation 1910 uses the components and pose information 1908 to determine a flight path 1912. The flight path is then provided for use with a second phase inspection operation 1914, the output of which is inspection images 1916 of at least a subset of components of the structure.

[0224] FIGS. 20A-B are illustrations of example columns of a flight path used during an inspection phase for semantic 3D scan, it being recognized that columns are merely one option of trajectory for a flight path in accordance with the implementations of this disclosure. Referring first to FIG. 20A, a UAV 2000 approaches a structure 2002 (e.g., a powerline transmission tower). The UAV 2000 and the structure 2002 may, for example,respectively be the UAV 1702 and the structure 1704 shown in FIG. 17. The UAV 2000 is configured to navigate around the structure to perform an inspection of the structure according to a flight path including four columns 2004, 2006, 2008, and 2010, which generally are located at comers of a rectangular boundary around the structure 2002. The UAV 2000 first navigates at a first altitude to the column 2004 and descends at the location of the column 2004 to capture one or more images at one or more capture points associated with the column 2004. The UAV 2000, after descending to a final capture point associated with the column 2004 and capturing one or more images at the final capture point, ascends to a navigation height and flies to the column 2006. The UAV 2000 then proceeds to descend to associated capture points and capture applicable images thereat before ascending to fly to the column 2008 and eventually the column 2010, in which the UAV 2000 operates as described above to capture images at applicable capture points thereof. The UAV 2000, after capturing a final image at a capture point of the column 2010, is deemed to have completed the flight path and accordingly returns to a dock thereof or another location.

[0225] Referring next to FIG. 20B, a UAV 2012 approaches a structure 2014 (e.g., a powerline transmission tower). The UAV 2012 and the structure 2014 may, for example, respectively be the UAV 1702 and the structure 1704 shown in FIG. 17. The UAV 2012 is configured to navigate around the structure to perform an inspection of the structure according to a flight path including two columns 2016 and 2018, which are generally located at opposing diagonal corners of a boundary of the structure 2014. The UAV 2012 first navigates at a first altitude to the column 2016 and descends at the location of the column 2016 to capture one or more images at one or more capture points associated with the column 2016. The UAV 2012, after descending to a final capture point associated with the column 2016 and capturing one or more images at the final capture point, ascends to a navigation height and flies to the column 2018. The UAV 2012 again descends at the location of the column 2018 to capture one or more images at each capture point associated with the column 2018. The UAV 2012, after capturing a final image at a capture point of the column 2018, is deemed to have completed the flight path and accordingly returns to a dock thereof or another location.

[0226] To further describe some implementations in greater detail, reference is next made to examples of techniques for semantic 3D scan for multi-phase structure inspection. FIG. 21 is a flowchart of an example of a technique 2100 for multi-phase semantic 3D scan for structure inspection. The technique 2100 can be executed using computing devices, such as the systems, hardware, and software described with respect to FIGS. 1-5 and 17-20B. Thetechnique 2100 can be performed, for example, by executing a machine-readable program or other computer-executable instructions, such as routines, instructions, programs, or other code. The steps, or operations, of the technique 2100, or another technique, method, process, or algorithm described in connection with the implementations disclosed herein can be implemented directly in hardware, firmware, software executed by hardware, circuitry, or a combination thereof.

[0227] For simplicity of explanation, the technique 2100 is depicted and described herein as a series of steps or operations. However, the steps or operations in accordance with this disclosure can occur in various orders and / or concurrently. Additionally, other steps or operations not presented and described herein may be used. Furthermore, not all illustrated steps or operations may be required to implement a technique in accordance with the disclosed subject matter.

[0228] At 2102, user input associated with a structure to inspect is obtained. The user input corresponds to one or more configurations for the inspection of the structure. For example, the user input may correspond to one or more of an object of interest, a traversal height, a flight distance, a maximum speed, an exploration radius, a gimbal angle, or a column path, wherein the user input is used to perform the first phase inspection. The user input may, for example, be obtained from a user device in communication (e.g., wireless communication) with a UAV to use to perform the inspection.

[0229] At 2104, a first phase inspection of the structure is performed using the UAV to determine a semantic understanding of components of the structure and pose information of the components. Performing the first phase inspection of the structure includes navigating the UAV to a distance from the structure such that all of the structure is within a field of view of a camera of the UAV. Performing the first phase inspection of the structure further includes detecting the components using a camera of the unmanned aerial vehicle and storing data indicative of the detected components on two-dimensional images associated with associated poses of the camera. That is, once the structure is centered within the field of view of the camera of the UAV, and based on a determination that the structure covers a threshold percentage of the field of view, one or more images of the structure are captured using the camera. Thereafter, the components of the structure are detected within the one or more images, and data indicative of the detected components is stored for later use during the multi-phase inspection. For example, the components may be detected within the one or more images using a computer vision technique or multiple such techniques, such as one or more of object detection, object recognition, or image segmentation. Detecting the componentsmay further, for example, include triangulating, using the UAV, locations of the components, such as to determine unique locations of the components in 3D space.

[0230] At 2106, a flight path is determined based on the components and the pose information. The flight path indicates capture points and camera poses associated with the capture points. For example, determining the flight path may include determining a number of columns for the unmanned aerial vehicle to vertically navigate, wherein each column of the number of columns includes one or more of the capture points. In non-limiting examples, the number of columns may be two or four.

[0231] At 2108, a second phase inspection of the structure is performed using the UAV according to the flight path. The second phase inspection may be performed for all components detected within the one or more images captured during the first phase inspection or for a subset of those components. Performing the second phase inspection may include navigating the UAV in a particular manner based on the number of columns of the flight path. For example, where the number of columns is four columns, the four columns form a rectangular boundary surrounding the structure, and performing the second phase inspection of the subset of the components according to the flight path may include navigating the UAV about the rectangular boundary including ascending to a traversal height while moving between ones of the four columns. In another example, where the number of columns is two, the two columns are at diagonally opposing locations about the structure, and performing the second phase inspection of the subset of the components according to the flight path may include navigating the UAV between the two columns over the structure. In either case, performing the second phase inspection according to the flight path includes, while the UAV is at a capture point of the capture points, aiming a camera of the UAV at a component of the components according to a camera pose of the camera poses, and capturing, using the aimed camera, an image of the component. For example, aiming the camera at the component according to the camera pose may include continuously attempting to detect the component within a video feed captured using the camera until the component is centered in images of the video feed.

[0232] At 2110, inspection data produced based on the second phase inspection is output. The inspection data may, for example, include one or more images captured during the second phase inspection. In some cases, the one or more images of the inspection data may be labeled with information associated with one or both of the structure or the flight path. For example, labeling the one or more images of the inspection data may include introducing metadata (e.g., EXIF metadata) on files of the one or more images. Alternatively, labeling theone or more images of the inspection data may include moving files of the one or more images into folders of a storage system (e.g., at the UAV, user device, or server), such that the labeling corresponds to the corresponding images being labeled according to a name of their respective folders.

[0233] In some implementations, the exploration phase and inspection phase may be collectively referred to as a first inspection, and a second inspection using the UAV may be performed or at least initiated while the first inspection remains in progress. For example, an exploration phase of a second inspection may be interleaved with the inspection phase of the first inspection. In such a case, the exploration of next structure components may be performed at the same time as the inspection of the current structure components.

[0234] The implementations of this disclosure can be described in terms of functional block components and various processing operations. Such functional block components can be realized by a number of hardware or software components that perform the specified functions. For example, the disclosed implementations can employ various integrated circuit components (e.g., memory elements, processing elements, logic elements, look-up tables, and the like), which can carry out a variety of functions under the control of one or more microprocessors or other control devices.

[0235] Similarly, where the elements of the disclosed implementations are implemented using software programming or software elements, the systems and techniques can be implemented with a programming or scripting language, such as C, C++, Java, JavaScript, assembler, or the like, with the various algorithms being implemented with a combination of data structures, objects, processes, routines, or other programming elements.

[0236] Functional aspects can be implemented in algorithms that execute on one or more processors. Furthermore, the implementations of the systems and techniques disclosed herein could employ a number of conventional techniques for electronics configuration, signal processing or control, data processing, and the like. The words “mechanism” and “component” are used broadly and are not limited to mechanical or physical implementations, but can include software routines in conjunction with processors, etc. Likewise, the terms “system” or “tool” as used herein and in the figures, but in any event based on their context, may be understood as corresponding to a functional unit implemented using software, hardware (e.g., an integrated circuit, such as an ASIC), or a combination of software and hardware. In certain contexts, such systems or mechanisms may be understood to be a processor-implemented software system or processor-implemented software mechanism that is part of or callable by an executable program, which may itself be whollyor partly composed of such linked systems or mechanisms.

[0237] Implementations or portions of implementations of the above disclosure can take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium can be a device that can, for example, tangibly contain, store, communicate, or transport a program or data structure for use by or in connection with a processor. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device.

[0238] Other suitable mediums are also available. Such computer-usable or computer- readable media can be referred to as non-transitory memory or media, and can include volatile memory or non-volatile memory that can change over time. A memory of an apparatus described herein, unless otherwise specified, does not have to be physically contained by the apparatus, but is one that can be accessed remotely by the apparatus, and does not have to be contiguous with other memory that might be physically contained by the apparatus.

[0239] While the disclosure has been described in connection with certain implementations, it is to be understood that the disclosure is not to be limited to the disclosed implementations but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures as is permitted under the law.

Claims

What is claimed is:

1. A method, comprising: obtaining images captured using a camera of an unmanned aerial vehicle during an exploration inspection of a structure; determining components of the structure based on the images and a taxonomy associated with the structure; generating a visual representation of the components; and outputting the visual representation of the components to a user device in communication with the unmanned aerial vehicle to enable selections of ones of the components for further inspection using the unmanned aerial vehicle.

2. The method of claim 1, wherein determining the components of the structure based on the images and the taxonomy associated with the structure comprises: performing a computer vision process to detect the components within the images; and identifying the components by object type using the taxonomy.

3. The method of claim 2, wherein identifying the components by the object type using the taxonomy comprises: using a machine learning model to process the detected components against the taxonomy.

4. The method of claim 2, comprising: updating the taxonomy based on the detected components using a machine learning model.

5. The method of claim 2, wherein the computer vision process includes at least one of object detection or image segmentation.

6. The method of claim 1, wherein the visual representation of the components includes a three-dimensional graphical representation of the structure and generating the visual representation of the components comprises: labeling respective portions of the three-dimensional graphical representation of thestructure according to information associated with the components.

7. The method of claim 6, wherein labeling the respective portions of the three- dimensional graphical representation of the structure according to information associated with the components comprises: annotating a portion of the three-dimensional graphical representation corresponding to a component with an unfavorable status to indicate the unfavorable status.

8. The method of claim 1, wherein the visual representation of the components includes a hierarchical text representation of the structure and generating the visual representation of the components comprises: generating the hierarchical text representation of the structure according to an arrangement of the components within the taxonomy.

9. The method of claim 1, comprising: obtaining, from the user device, user input indicating the selections of the ones of the components within a graphical user interface within which the visual representation of the components is output for display.

10. An unmanned aerial vehicle, comprising: one or more cameras; one or more memories; and one or more processors configured to execute instructions stored in the one or more memories to: capture one or more images of a structure using the one or more cameras; determine components of the structure based on the images and a taxonomy associated with the structure; and output a visual representation of the components to a user device to enable selections of ones of the components for inspection.

11. The unmanned aerial vehicle of claim 10, wherein the components are detected based on a computer vision process performed against the one or more images and identified by object type using the taxonomy.

12. The unmanned aerial vehicle of claim 10, wherein the one or more processors are configured to execute the instructions to: generate the visual representation of the components based on the determination of the components.

13. The unmanned aerial vehicle of claim 10, wherein the visual representation includes one or both of a three-dimensional graphical representation of the structure or a hierarchical text representation of the structure.

14. The unmanned aerial vehicle of claim 10, wherein user input indicating the selections of the ones of the components is obtained from the user device to configure the unmanned aerial vehicle to perform an inspection of the selected ones of the components.

15. A system, comprising: an unmanned aerial vehicle; and a user device in communication with the unmanned aerial vehicle, wherein the unmanned aerial vehicle is configured to: capture images of a structure during an exploration inspection of the structure; determine components of the structure based on the images and a taxonomy associated with the structure; and output, to the user device, a visual representation of the components to enable selections of ones of the components for further inspection.

16. The system of claim 15, wherein the components are determined based on a detection of the components by a computer vision process performed against the images and an identification of object types of the detected components within the taxonomy.

17. The system of claim 15, wherein the user device is configured to: render a graphical user interface that outputs the visual representation of the components for display; and obtain, via the graphical user interface, user input indicating the selections of the ones of the components.

18. The system of claim 17, wherein the unmanned aerial vehicle is configured to:perform a further inspection of the ones of the components based on the user input.

19. The system of claim 15, wherein the unmanned aerial vehicle is configured to: obtain a three-dimensional graphical representation of the structure; and generate the visual representation of the components by labeling respective portions of the three-dimensional graphical representation of the structure.

20. The system of claim 15, wherein the unmanned aerial vehicle is configured to: generate the visual representation of the components as a hierarchical text representation of the structure according to an arrangement of the components within the taxonomy.

21. A method, comprising: obtaining images captured during an unmanned aerial vehicle-based exploration inspection of a structure; encoding, in each pixel value that corresponds to the structure within the images, identifiers of the structure, a component of the structure depicted using the pixel value, and a location of the component; segmenting pixel values of the images into polygons according to the encoded identifiers; and storing data indicative of the polygons for use in a further inspection of the structure.

22. The method of claim 21, comprising: obtaining a three-dimensional graphical representation of the structure; and rendering, according to the encoded identifiers, the three-dimensional graphical representation of the structure using shaders that visually distinguish each component of the structure, wherein the data indicative of the polygons identifies respective ones of the shaders.

23. The method of claim 22, wherein obtaining the three-dimensional graphical representation of the one or more structures comprises: generating the three-dimensional graphical representation based on the images using a ray-based optimization technique.

24. The method of claim 22, wherein rendering the three-dimensional graphical representation of the structure using the shaders for each component of the structure according to the encoded identifiers comprises: generating a UV map of the structure according to the shaders; and rendering the three-dimensional graphical representation of the structure based on the UV map.

25. The method of claim 24, wherein each pixel value is represented by coordinates of the UV map.

26. The method of claim 22, wherein rendering the three-dimensional graphical representation of the structure using the shaders for each component of the structure according to the encoded identifiers comprises: rendering different surfaces of a component of the structure within the three- dimensional graphical representation using different ones of the shaders.

27. The method of claim 21, wherein segmenting the pixel values of the images into the polygons according to the encoded identifiers comprises: performing a panoptic segmentation process against the images to segment the pixel values into vector annotations corresponding to the polygons.

28. The method of claim 21, comprising: for each component of the structure, determining a location of the component using a visual positioning system of the unmanned aerial vehicle and pose information of the unmanned aerial vehicle.

29. The method of claim 21, comprising: obtaining a query for images depicting the structure of one or more structures; and using the stored data to indicate the images in response to the query.

30. An unmanned aerial vehicle, comprising: one or more cameras; one or more memories; and one or more processors configured to execute instructions stored in the one or morememories to: capture images of a structure using the one or more cameras; render a three-dimensional graphical representation of the structure with shaders visually distinguishing components of the structure by encoding, in each pixel value that corresponds to the structure within the images, identifiers of the structure, a component of the structure depicted using the pixel value, and a location of the component; and storing data indicative of polygons associated with pixel values of the images for use in a further inspection of the structure.

31. The unmanned aerial vehicle of claim 30, wherein the one or more processors are configured to execute the instructions to: segment the pixel values of the images into the polygons according to the encoded identifiers.

32. The unmanned aerial vehicle of claim 31, wherein, to segment the pixel values of the images into the polygons according to the encoded identifiers, the one or more processors are configured to execute the instructions to: perform a panoptic segmentation process against the images to segment the pixel values into vector annotations corresponding to the polygons.

33. The unmanned aerial vehicle of claim 30, wherein, to render the three- dimensional graphical representation of the structure with the shaders, the one or more processors are configured to execute the instructions to: generate a UV map of the structure according to the shaders; and render the three-dimensional graphical representation of the structure based on the UV map.

34. The unmanned aerial vehicle of claim 33, wherein, to render the three- dimensional graphical representation of the structure with the shaders, the one or more processors are configured to execute the instructions to: associate the pixel value with coordinates of the UV map of the structure.

35. A system, comprising:an unmanned aerial vehicle; and a user device in communication with the unmanned aerial vehicle, wherein the unmanned aerial vehicle is configured to: render a three-dimensional graphical representation of a structure with shaders visually distinguishing components of the structure by encoding, in each pixel value that corresponds to the structure within images captured of the structure, identifiers of the structure, a component of the structure depicted using the pixel value, and a location of the component; and cause an output of data associated with the rendered three-dimensional graphical representation of the structure at the user device.

36. The system of claim 35, wherein the unmanned aerial vehicle is configured to: capture the images; and obtain the three-dimensional graphical representation of the structure.

37. The system of claim 36, wherein, to obtain the three-dimensional graphical representation of the structure, the unmanned aerial vehicle is configured to: generate the three-dimensional graphical representation based on the images using graph clustering.

38. The system of claim 35, wherein the unmanned aerial vehicle is configured to: segment pixel values of the images into polygons according to the encoded identifiers; and store data indicative of the polygons for use in a further inspection of the structure.

39. The system of claim 38, wherein, to segment the pixel values of the images into the polygons according to the encoded identifiers, the unmanned aerial vehicle is configured to: perform a panoptic segmentation process against the images to segment the pixel values into vector annotations corresponding to the polygons.

40. The system of claim 38, wherein the data indicative of the polygons identifies respective ones of the shaders.

41. A method, comprising: performing, using an unmanned aerial vehicle, a first phase inspection of a structure to determine a semantic understanding of components associated with the structure and pose information of the components; determining, based on the semantic understanding of the components and the pose information, a flight path indicating capture points and camera poses associated with the capture points; and performing, using the unmanned aerial vehicle, a second phase inspection of a subset of the components according to the flight path.

42. The method of claim 41, performing the first phase inspection of the structure to determine the semantic understanding of the components and the pose information comprises: navigating the unmanned aerial vehicle to a distance from the structure such that all of the structure is within a field of view of a camera of the unmanned aerial vehicle.

43. The method of claim 41, wherein performing the first phase inspection of the structure to determine the semantic understanding of the components and the pose information comprises: detecting the components using a camera of the unmanned aerial vehicle; and storing data indicative of the detected components on two-dimensional images associated with associated poses of the camera.

44. The method of claim 43, wherein detecting the components comprises: triangulating, using the unmanned aerial vehicle, locations of the components.

45. The method of claim 41, wherein determining the flight path indicating the capture points and the camera poses comprises: determining a number of columns for the unmanned aerial vehicle to vertically navigate, wherein each column of the number of columns includes one or more of the capture points.

46. The method of claim 45, wherein the number of columns is four columns, the four columns form a rectangular boundary surrounding the structure, and performing thesecond phase inspection of the subset of the components according to the flight path comprises: navigating the unmanned aerial vehicle about the rectangular boundary including ascending to a traversal height while moving between ones of the four columns.

47. The method of claim 45, wherein the number of columns is two columns, the two columns are at diagonally opposing locations about the structure, and performing the second phase inspection of the subset of the components according to the flight path comprises: navigating the unmanned aerial vehicle between the two columns over the structure.

48. The method of claim 41, wherein performing the second phase inspection of the subset of the components according to the flight path comprises: while the unmanned aerial vehicle is at a capture point of the capture points, aiming a camera of the unmanned aerial vehicle at a component of the components according to a camera pose of the camera poses; and capturing, using the aimed camera, an image of the component.

49. The method of claim 48, wherein aiming the camera at the component according to the camera pose comprises: continuously attempting to detect the component within a video feed captured using the camera until the component is centered in images of the video feed.

50. The method of claim 48, comprising: labeling the image with information associated with one or both of the structure or the flight path.

51. The method of claim 41, comprising: obtaining user input corresponding to one or more of an object of interest, a traversal height, a flight distance, a maximum speed, an exploration radius, a gimbal angle, or a column path, wherein the user input is used to perform the first phase inspection.

52. An unmanned aerial vehicle, comprising:one or more cameras; one or more memories; and one or more processors configured to execute instructions stored in the one or more memories to: perform a first phase inspection of a structure to determine, based on one or more images captured using the one or more cameras, a semantic understanding of components associated with the structure and pose information of the components; and perform a second phase inspection of a subset of the components according to a flight path that is based on the semantic understanding of the components and the pose information.

53. The unmanned aerial vehicle of claim 52, wherein, to perform the first phase inspection of the structure to determine the semantic understanding of the components and the pose information, the one or more processors are configured to execute the instructions to: capture the one or more images while all of the structure remains within a field of view of the one or more cameras; and perform image segmentation to detect the components within the one or more images.

54. The unmanned aerial vehicle of claim 53, wherein, to detect the components within the one or more images, the one or more processors are configured to execute the instructions to: triangulate locations of the components to determine unique locations in three- dimensional space of the components.

55. The unmanned aerial vehicle of claim 52, wherein, to perform the second phase inspection of the subset of the components according to the flight path, the one or more processors are configured to execute the instructions to: aim the one or more cameras at a component; and capture, using the aimed one or more cameras, an image of the component.

56. The unmanned aerial vehicle of claim 52, wherein the one or more processors are configured to execute the instructions to:determine the flight path based on the semantic understanding of the components and the pose information, wherein the flight path indicates capture points and camera poses associated with the capture points, and wherein the capture points are arranged into a number of columns for the unmanned aerial vehicle to vertically navigate.

57. A system, comprising: an unmanned aerial vehicle; and a user device in communication with the unmanned aerial vehicle, wherein the unmanned aerial vehicle is configured to: perform a first phase inspection of a structure according to user input obtained from the user device to determine a semantic understanding of components associated with the structure and pose information of the components; determine, based on the semantic understanding of the components and the pose information, a flight path indicating a number of columns for the unmanned aerial vehicle to vertically navigate, wherein each column of the number of columns includes one or more capture points each associated with one or more of the components; and perform a second phase inspection according to the flight path.

58. The system of claim 57, wherein the first phase inspection is performed while all of the structure is within a field of view of a camera of the unmanned aerial vehicle and the components are determined based on images captured using the camera.

59. The system of claim 58, wherein the camera is aimed at a component of the one or more of the components according to a camera pose associated with a respective capture point of the one or more capture points to capture an image of the component during the second phase inspection.

60. The system of claim 57, wherein, where the number of columns is four columns, the four columns form a rectangular boundary surrounding the structure and the unmanned aerial vehicle navigates about the rectangular boundary including ascending to a traversal height while moving between ones of the four columns, and wherein, where the number of columns is two columns, the two columns are at diagonally opposing locations about the structure and the unmanned aerial vehicle navigatesbetween the two columns over the structure.

Images